Sound and Video Anthology: Program Notes

This sound anthology has been conceived as a companion piece to our special issue on “Musical Interactivity in Human–AI and AI–AI Partnerships.” Most of the pieces in this collection directly relate to articles from this special issue. We hope that this provides a more complete experience, offering a deeper understanding and appreciation of the roles and dynamics of interactions between the different actors involved.

Within this diverse selection of works, you will encounter a broad spectrum of artistic expressions. This diversity extends beyond musical styles to encompass a wide range of practices, including composition, improvisation, live coding, and DJ set production, as well as a wide range of modes of interactions between musicians and artificial intelligence (AI) systems. Each piece serves as a unique example of how AI's role in music creation has evolved beyond the conventional boundaries of a simple tool, with AI systems actively participating in the creative process.

To provide you with deeper insights into these works, the descriptions below are authored by the composers themselves. These descriptions vary in length, offering insights into each composer's artistic vision and the essence of their compositions.

“Between the Lines” (2018) is based on a machine learning system developed in collaboration with Bob Sturm: FolkRNN. This is a recurrent neural network that crafts melodies: We first trained the system on 23,000+ melodic transcriptions of traditional music primarily from Ireland and the UK. To compose this piece we added a second stage where we focused its training on melodies extracted from over 350 English Renaissance vocal pieces by Thomas Tallis and his contemporaries. Training the system in this manner makes it learn first about the music representation, and then about melodies typical of English vocal Renaissance music.

To compose this piece, I prompted the system with the opening notes from Tallis's “Miserere nostri,” yielding continuations of these opening notes. Sifting among the many melodies generated, I constructed a short, four-part musical phrase that forms the core of the piece, which is repeated throughout the piece by the melodic instruments. By reinterpreting the rhythms of each voice, in a manner that echoes the mensuration canons used by Tallis in his “Miserere” setting, the melodies slide against one other in each repetition. The piano part was also generated algorithmically, though not using deep learning. The algorithm uses the intervallic structure of the same opening notes—alternating major and minor thirds—and extends this structure over the whole piano range. This piano part provides a type of commentary on the repeated phrase.

The piece exemplifies “human-in-the-loop” composition, with AI in an assistive capacity. FolkRNN (after re-training) provided raw material for a composition process allowing me to experiment with both the materials and processes of composing. Crafting an algorithmic generator (implemented in SuperCollider) for the piano part was part of that process. Contending with the way FolkRNN generates material as well as the traces of historical musical practices encapsulated in the system altered my composition process and outcome in unexpected ways.

“Between the Lines” was premièred by the New Music Players at the Attenborough Centre for the Creative Arts (University of Sussex, Brighton, UK), 8 February 2018.

Track Duration: 6:19

Oded Ben-Tal is a composer and researcher working at the intersection of music, computing, and cognition. His compositions range from purely acoustic pieces to interactive, live electronic pieces, and multimedia work. In recent years he has been particularly interested in the interaction between human and computational creativities. Together with Bob Sturm he developed research applying deep learning to folk musics and interrogating the creative capacity of the resulting generative system within the folk tradition as well as outside it.

He is also using AI-inspired approaches in the domain of interactive, live electronic music. Machine listening techniques combined with algorithmically steered processes open space for musical dialogue, in real time, between human performers and computer counterparts.

In 2022, he launched the Datasounds, Datasets, and Datasense research network, which aims to identify core questions that will drive forward the next phase in data-rich music research, focused in particular on creative music making—stemming from his own compositional interests in relating human and machine creativities and broadening the scope to consider the implications as well as applications of computational means used to make music, understand it, and engage with it.

Joëlle Léandre has been in artistic residency at Institut de Recherche et Coordination Acoustique/Musique (IRCAM) from June 2022 to June 2023, experimenting with interaction technologies for AI–musician improvisation developed by the REACH project, in particular the Omax and Somax2 autonomous-agent environments. With Gérard Assayag, Marco Fiorini, and Mikhail Malt (the “Who/Men”) Léandre has thoroughly explored the concept of human–AI cocreativity fostered by REACH, with a great achievement: the REACHing OUT concert during the IRCAM Manifeste Festival in June featuring Léandre and The Who/Men, and the American experimental rock band Horse Lord, who went through a residency as well and performed with the Who/Men and Somax2.

Track Duration: 7:17

Joëlle Léandre received the Lifetime Achievement Award at the Vision Festival on 13 June 2023 in New York. As noted in the program for that festival, as “a progenitor of European improvisation, bassist Joëlle Léandre's ongoing career is an integral story in 20th and 21st century music. Léandre has amassed over 200 recordings, including numerous collaborations with fellow improvising luminaries such as Marilyn Crispell, Anthony Braxton, Nicole Mitchell, and many more. From early in her career, the caliber of her talent attracted attention. Léandre has performed with Leonard Bernstein, Pierre Boulez, and Giacinto Scelsi, as well as with Merce Cunningham and John Cage. The latter two composed Ryoanji for Leandre specifically. Over 40 composers have dedicated pieces to her.”

AI is the magic of the present. In our daily rituals we ask the computer our inner questions and respond to its recommendations. Computer music algorithms with names like the “oracle” are summoned by performers like shamans. In “Prestidigitation,” written in collaboration with percussionist Maxime Echardour, the performer enters a darkened stage as a spotlight reveals an empty frame. One by one, he pulls from the dark an array of objects and materials of far-flung origins—seashells, nutshells, bells, chimes—and assembles them into a growing sculptural installation. He brings each object to life like a puppeteer, examining its delicate texture around a 3-D microphone. He completes the sculpture with a frame drum with which he brings together the entire assemblage into composite gestures of growing density. Both minute and explosive, the sounds are captured and projected to the listeners in three dimensions, situating them in the middle of the microtheater. The totemic loudspeaker in the middle of the room at first amplifies the performer's gestures, then gradually extends them into improvisations, blurring the boundary between the real and the virtual. From within the detailed score, the performer responds with brief improvisatory passages of his own, to which the loudspeaker multiplies its unpredictable feedback. The ritual closes as the percussionist leaves the stage and the loudspeaker takes over with a final solo improvisation, but the oracle's answers are inconclusive.

“Prestidigitation” integrates a critical approach to machine learning embedded within a technical assemblage encompassing performer, instruments, microphone, computer, loudspeaker, and audience. The creative process balances the “material engagement” of composer and performer with computer improvisation and embodied spatial sound synthesis in a “dance of agency” between human and nonhuman actors. The results offer a unique juxtaposition of a fully composed score with passages of spatially situated improvisation made possible through creative engagement with AI. The work was produced for the project Permeable Interdisciplinary: Algorithmic Composition, Subverted, part of the research program Music and Artificial Intelligence: Building Critical Interdisciplinary Studies (MusAI), led by Georgina Born and funded by the Advanced Grants scheme of the European Research Council.

This live concert recording of “Prestidigitation” features a binaural rendering of 3-D spatial electronics that is improved when heard with high-quality headphones.

Track Duration: 20:18

Aaron Einbond’s work explores the intersection of instrumental music, field recording, sound installation, and interactive technology questioning relationships between instrument, loudspeaker, listener, space, and place. The record label All That Dust recently released his portrait album Cosmologies, performed by cellist Séverine Ballon, pianist Alvise Sinivia, and the Riot Ensemble. His album Cities features collaborations with ensemble Yarn/Wire and Matilde Meireles, and his portrait album Without Words was recorded by Ensemble Dal Niente. Other recent and upcoming collaborators include the ensembles United Instruments of Lucilin, soundinitiative, and the Opera Lab Berlin; soloists Maxime Echardour, Samuel Stoll, and Marco Fusi; and festivals Présences, November Music, and the Académie du Festival d'Aix-en-Provence. He has received a John Simon Guggenheim Memorial Foundation Fellowship, a Giga-Hertz Advancement Award, and Artistic Research residencies at IRCAM and Zentrum für Kunst und Medien (ZKM). He teaches at City, University of London, and is Co-Artistic Director of Qubit New Music in New York. He has taught at Columbia University, the University of Huddersfield, and Harvard University, and studied at Harvard University, the University of Cambridge, the University of California, Berkeley, and IRCAM with teachers including Mario Davidovsky, Julian Anderson, and Philippe Leroux.

“Bias II,” for piano and interactive music system, explores distributed and posthuman notions of musical creativity, enabled through processes of datafication and machine learning. During its interactions with different pianists, the computer music system collects data pertaining to the interpretative choices made by performers—specifically, the way performers navigate a set of seven clusters, each consisting of a variable number of timbrally similar musical actions. Based on predictions made by a recurrent neural network using these data, the computer codetermines the form of the performance, by choosing to follow the musician or propose different sound material. Historical data, collected by the computer music system in past performances, influence its future behavior, turning the work into an open-ended cocreative process that is distributed in space and time.

At the time of writing, “Bias II” has been performed by, and trained on data from, pianists Magda Mayas and Xenia Pestova-Bennett. The piece was commissioned by and produced at ZKM | Hertz Lab. Research on this work was funded by the ERC advanced grant MusAI—Music and Artificial Intelligence: Building Critical Interdisciplinary Studies. The recording of this piece was made at ZKM Karlsruhe in the context of the Giga-Hertz Award. Performer: Magda Mayas; recording engineer: Sebastian Schottke.

Track Duration: 14:49

Artemi-Maria Gioti is a composer and artistic researcher working in the field of artificial intelligence. Her compositions include works for acoustic instruments and interactive electronics, musical robotics, and participatory performances, and they have been performed across Europe and in Canada and the USA. Her research explores the transformative potential of new technologies for musical thinking and seeks to redefine notions of authorship, performership, and the construct of the musical work. Interactivity is a central focus of her art and research, which views the musical work as the product of collaborative and distributed human–human and human–computer cocreativity. She studied composition, electroacoustic composition, and computer music at the University of Macedonia, Greece, the University of Music and Performing Arts Vienna, and the University of Music and Performing Arts Graz. She holds a doctoral degree in Music Composition from the last of these institutions. She is currently lecturer at the University of Music Carl Maria von Weber, Dresden, and a research fellow in Music and Artificial Intelligence at University College London, working on the ERC project MusAI, led by Georgina Born.

“Conversational Learning” explores live coding liveness within the machine learning process (data collection, training, and validation), focusing on how real-time training of a machine learning algorithm can be sonically exposed. It is based on a rule-learning algorithm that automatically produces new synthesizer presets out of a small, labeled database. The algorithm is designed with only two parameters controlling how close the newly created presets can be to those originally present in the training data. The learning process happens midperformance, tweaking the algorithm parameters on the fly. Then, the different learned models unfold the piece in “conversation” with the performer.

“Conversational Learning” was performed at Immersed in Code, part of the International Conference on Live Coding, on Friday 21 April 2023 in the former Pieter Baan Centrum (Almere, The Netherlands).

Track Duration: 24:11

Iván Paz has a background in physics, mathematics, and computer science. He likes to investigate how the intersection between science, arts, and humanities offers new creative possibilities---in particular, how we interact with machine learning algorithms and the use of code as a real-time interface. His works use custom-developed software and electronics. He has collaborated with universities, designed interactive systems for festivals, and presented works and performances across the Americas and Europe. He is part of the live-coding communities of Mexico and Barcelona and writes in third person when necessary.

“Emergent Rhythm” is an audiovisual DJ performance using real-time AI audio generation models. Artist/DJ Tokui manipulates multiple models on stage to spontaneously generate rhythms and melodies. He then combines and mixes the generated audio loops to create musical developments.

This work is the third installment of his AI DJ Project, a series of attempts to explore the future of AI-based music production and live DJ performance. The project began as a back-to-back DJ session with the artist and an AI DJ system, then evolved into a live performance using various AI symbolic (MIDI) music generation models on stage. This time, the artist tries to use AI audio synthesis models in real time and faces unprecedented challenges. Everything heard during this performance is purely AI-generated sound.

Tokui adapted generative adversarial network (GAN) models, a default AI-image synthesis model, for audio synthesis. StyleGAN models trained on spectrograms of various sound materials generate spectrograms, and vocoder GAN models convert them into audio files. The process is faster than real time.

Aligning with the visual theme, Tokui extracted audio loops from various natural and human-made environmental sounds as well as music and used them as training data for audio generation. In addition, the artist uses real-time timbre transfer effects using Neutone, an AI audio plug-in he helped develop and release for other artists. The AI plug-in converts incoming audio into various singing voices, such as Japanese Buddhist chants. This highlights the diversity and commonality within the human cultural heritage.

From a DJ session, in which existing songs are selected and mixed, to a live performance that generates songs spontaneously and develops them in response to the audience's reactions: In this performance, the human DJ is expected to become an AJ, or “AI jockey,” rather than a disc jockey, taming and riding the AI-generated audio stream in real time. With the unique morphing sounds created by AI and the new degrees of freedom that AI allows, the AI jockey will offer audiences a unique, even otherworldly sonic experience. “Emergent Rhythm” was recorded live at MUTEK Japan (Tokyo) on 8 December 2022.

Track Duration: 33:10

Nao Tokui is an artist, researcher, and DJ. He is also the founder of Qosmo, an AI creative studio, and Neutone, an AI audio tech company. He received his PhD from the University of Tokyo for his human–computer interaction and AI research. He has been exploring the potential expansion of human creativity through the use of AI. His works have been exhibited at New York's Museum of Modern Art (MoMA), Barbican Centre (London), InterCommunication Center (Tokyo), and more. His first book on AI and creativity won the Okawa Publishing Award 2021. He is currently an adjunct associate professor at Keio University.