A Piano for Visuals: Affordances of Scan Processing Instruments

Scan processing is an analog technique used during the 1970s and 1980s to animate, manipulate, and synthesize video imagery that is known for its specific visual effect of warping the image’s scan lines (see Fig. 1). One of the first scan processors available commercially was the Scanimate system, patented by Computer Image Corporation in 1969 [1]. The Scanimate was intended chiefly to replace time-consuming methods of stop-motion animation with an automated, electronic workflow. Typically, two- and three-dimensional artworks, such as television programs’ logos, were scanned by a video camera and animated in real time by analog computer circuitry [2]. Originally, the Rutt/Etra Video Synthesizer was marketed to video production studios and schools as an affordable production tool [3] that offered fewer functions at a far smaller size and lower cost compared to the Scanimate, but its 1973 list price of US$12,000 was still the equivalent of a luxury automobile. Those on more limited budgets and with more experimental goals—such as artists working in universities, video cooperatives, or community television stations—learned to construct their own production tools from parts intended for industrial, scientific, or military applications. The CalArts Videographics Lab system that Michael Scroggins built in the early 1980s is a key example of this approach [4].

While scan processors can be described as tools for motion graphics, in practice, their real time operation resembles playing electronic music synthesizers strongly. This is particularly evident in the context of live audiovisual performances, or in situations where direct physical interaction through movement, voice, or other sounds is critical to the results. In other situations, a scan processor’s operation may resemble a music composer’s calculated intentions more closely. The works of Benton C Bainbridge, Gary Hill, and Vibeke Sorensen surveyed in this article illustrate these three different methods. Such musician-like contexts allow us to draw parallels between instruments of sound and those of the moving image, and to augment existing accounts of scan processors based upon the history of visual media [5] with analyses rooted in concepts of electronic instrument design [6].

I assert that the unique visual results produced through scan processing are highly specific to combinations of context, instrument, and user encompassed by the design concept of affordances. Psychologist James J. Gibson uses this term to describe the unique sets of possibilities that arise within an environment for any particular animal with its own needs and capabilities [7]. Early adoption of the term within the field of human-computer interaction focused largely on what possibilities an object offers to its users [8], although it has been expanded to a more “ecological” awareness since then [9]. In contrast to a unidirectional offering, Karen Barad’s agential realism—inspired by observations from quantum mechanics—proposes a complex entanglement of intra-actions between the objects in our world. From this perspective, oppositional categories such as designer/user, player/ instrument, or performer/audience are seen as intentional and arbitrary agential cuts that serve specific purposes for the agents involved and in no way affect their entangled inseparability [10].

Scholars of music performance also recognize that the process of “musicking” [11] is neither simply a uni- nor a bidirectional relationship between player and instrument, but instead represents a multi-agential traversal of the psychological, social, aesthetic, and technological domains [12]. Similarly, speculative organologies of the instruments of both music and science propose an extension beyond general categorizations of an instrument’s functionality to discuss what sorts of social relationships and power structures the instrument affords [13]. The correspondences between all of these domains give rise to the affordances of a specific instrument used in a specific manner by a specific person in a specific context.

Even after scan processors’ animation role in film and television was supplanted by smaller and cheaper digital devices in the 1980s, the affordances related to these instruments still capture artists’ imagination and inspire re-creations in the analog and digital domains. In this article, I intend to identify some of the qualities from the past that make scan processing so compelling five decades later, and to investigate reenacting some of these historical affordances in the present.

My research prioritizes first-hand accounts of scan processing instruments’ designers and users, obtained largely through interviews that I conducted between 2021 and 2024. These interviews were framed as conversations between knowledgeable peers within the same field rather than as those between an ethnographer and his subject [14]. Additional research was done through an online survey designed to gather information from the video art community about their use of scan processing tools [15]. This survey was posted on several Facebook groups’ pages related to historical video art technologies: Video Circuits; Vector Synthesis; Signal Culture Alumni; and the now-defunct Experimental Television Center group, as well as groups that specialize in contemporary live video software, such as Pure Data, Max, Touch Designer, and Processing. The survey gathered 71 responses. Direct access to the majority of the historical instruments considered was not possible. Therefore, I have supplemented the interviews and survey with an array of technical and historical material: schematics; printed manuals and guides; recorded video demonstrations, notes, letters, and statements; interviews in print and video media; and audiovisual works that use the instruments. This article incorporates only a small slice of this vast amount of material.

Understanding scan processing’s affordances begins with understanding the materiality of the cathode ray tube (CRT). A CRT display employs a beam of electrons to illuminate phosphor within the face of a vacuum tube. This beam moves either in a predictable zigzag raster scan pattern or in an arbitrary random scan. Examples of both patterns can be seen in Fig. 2. In a typical raster video system, only the information that controls the beam’s luminance is sent to the display. This information is nestled between pulses that synchronize the horizontal and vertical scanning of the CRT’s raster with the timing of the input video signal.

A scan processor is an analog computer optimized to process these raster scanning signals. It strips the synchronization information from a standard video signal and uses it to generate its own horizontal and vertical control signals. These signals can then be manipulated freely with respect to size and position by summing them with or multiplying them by other signals, such as periodic waveforms from a function generator. The horizontal and vertical signals manipulated are finally sent together with the brightness information to a random scan CRT such as an oscilloscope. The result is an image on the display that can be animated to spin, roll, bounce, zoom forward, or disappear into the distance under both manual and automated control. The raster itself can also be warped, twisted, and tilted in ways that suggest a 3D object rather than a 2D rectangle. Any manipulation of the video raster produces signals that cannot be displayed on a normal television or video monitor. To record or broadcast a scan processor’s output, a video camera is aimed at its CRT display in a process known as rescanning.

In the research interviews and community survey, the technique of luma displacement stands out among users as a signature visual effect of scan processing. It involves shifting the individual scan lines proportionate to the brightness in that portion of the image. The result appears on our visual apparatus as a pseudo-3D image, with depth cues simulated by the line displacement, as seen in Fig. 3. This effect is used so extensively in the work of video artists Steina and Woody Vasulka that it has been dubbed “the Vasulka effect” [16]. Beyond its aesthetic effect, luma displacement also has a number of technical and scientific image processing applications discussed in the following section.

One inspiration for scan processing lies in the convolutions seen on a CRT display when its power is switched off and the raster collapses upon itself. In the late 1960s, artists such as Ture Sjölander and Bror Wikström [17] and Nam June Paik explored methods to create a similar effect to disrupt the seamless illusion of broadcast television imagery. Paik’s Raster Manipulation Unit (dubbed the “Wobbulator” later), designed with Shuya Abe in 1969, used electromagnets to modify the shape of the video raster continuously. Although this device was not a true scan processor, the Wobbulator’s visual results were highly influential in the design process of later instruments, such as the Rutt/Etra Video Synthesizer [18]. However, few media technologies arise solely within the creative field. Scan processing’s use for technical image analysis applications preceded its later use in animation and video art. In the following section, we consider the affordances in the use of one such technical scan processor—the VP-8 Image Analyzer—before we discuss the work of three different artists who work with the Rutt/Etra.

During the 1970s, companies such as Optical Electronics Inc. (OEI) and Interpretation Systems Inc. (ISI) offered analog computer equipment for three-dimensional imaging, with potential applications in enhancing images, detecting motion, and controlling quality, as well as medical, sonar, radar, and astronomy applications [19]. Some of these technical instruments found their way into artistic use. For example, the use of OEI modules might focus design time on instrument interfaces rather than on prototyping core circuitry [20] or help when constructing a “poor man’s Rutt/Etra” [21]. In the case of ISI’s VP-8 Image Analyzer, we find the affordances in a ready-made scientific instrument designed to produce a remarkable cultural artifact.

Since the 1970s, an image that the VP-8 produced has been invoked regularly to ascribe the appearance of scientific validity to religious speculation about the origins of the Shroud of Turin. ISI production engineer Peter M. Schumacher recounts using the VP-8’s isometric projection function to process a photograph of the Shroud in 1976 [22]. An isometric projection maps input video images to a relief presentation, where brightness is represented by elevation from the two-dimensional image plane [23] in precisely the same manner as the luma displacement technique described earlier. An isometric projection can be rotated and tilted manually to view the image from different angles, as seen in Fig. 4, for example, to analyze the reflectivity of terrain features captured by aerial and satellite photography. Schumacher’s 1976 rendering appears to depict in accurate three-dimensional detail the body of a man whom Schumacher alleges to be Jesus Christ [24].

Schumacher points out that the image on the VP-8 display is not a 3D representation of the photo’s original subject. It is only a plot of brightness variations within the scanned photograph, which fools our visual apparatus into seeing a three-dimensional object on screen. Schumacher then claims that no other photograph or artwork that he has analyzed with the VP-8 has produced similar effects, and that this visual effect’s uniqueness supports his personal belief in the Shroud’s supernatural qualities [25]. Although video artists also used luma displacement at the time to create 3D-like imagery of the human form, Schumacher does not take this into consideration.

My purpose in discussing this image is not to delve into theological debate but rather to consider it as a cultural artifact—in fact, one of the scan-processed images seen most widely in the world. Schumacher’s account of the Shroud scan illustrates how affordances that emerge through the VP-8 technology relative to Schumacher and his world view shape his intra-actions with the instrument—in the same way that any technology’s psychological, social, aesthetic, and material aspects mediate an artist’s vision during the creative process— to reach a result that is fundamentally inseparable from the agents involved. It is not an image of the Shroud, nor is it an image created by Schumacher, nor is the image realized with the VP-8. It is not solely any of these things at all, but rather a manifestation of their combined potentials as affordances.

The Rutt/Etra represents the overlap between the worlds of commercial motion graphics and video art. Steve Rutt worked as an electronics engineer, and his vision for the instrument was solidly aimed at the broadcast television production market [26], while Bill and Louise Etra were artists who dreamed of inventing an instrument that allowed one to play images in the same way that a pianist plays music. In 1999, Bill Etra remarked:

Among the video artists whom I interviewed, I have selected three to represent their distinct approaches to the Rutt/Etra as an instrument for moving images and the affordances that arise in their relationship with it: Vibeke Sorensen as an animated image composer; Gary Hill as a process-oriented visual artist, and Benton C Bainbridge as a live video performer.

Apart from transformations applied to images captured by a camera, the Rutt/Etra can also create camera-less, non-representational, and purely synthetic abstract imagery [28] derived from programmable electronic signals. For Vibeke Sorensen, working with the intricate mathematical relationships between these signals requires a fluency analogous to a composer’s understanding of sound. Sorensen refers to this technical language as “Synthesizerese” and details how the array of dots in her work Temple (Fig. 5), as well as the periodic “squashing” of their arrangement, are both derived from sine waves running at multiples of the video raster’s horizontal and vertical frequencies [29]. Sorensen has also developed methods to generate stereoscopic images with the Rutt/Etra—not through a ready-to-hand function in the machine but rather as a skilled music synthesist builds up a complex sound from basic waveforms. In this case, Sorensen uses square waves to alternate between two variations of the same image rapidly, each of which has a slightly different perspective effect applied [30].

For artists like Gary Hill, the Rutt/Etra’s real-time nature, combined with the ability to bring other types of information into it, affords the greatest possibilities. In Happenstance (part one of many parts) (Fig. 6), Hill manipulates blocks of written text through the Rutt/Etra into beautiful, curvilinear abstract forms, all synced tightly with the rhythm of his spoken words. For Hill, the Rutt/Etra’s ability to sculpt the images in real time complements the immediacy of the voice:

Beyond the voice, Hill explores the use of other external signals such as cameras and audio synthesizers, noting that “if you just have a camera in a room, you started thinking about that camera as a waveform generator” [32] using objects in the room as sources. Even four decades later, Hill still considers the complementarity between voice and live visual synthesis that he explored in Happenstance as inexhaustibly inspiring.

Benton C Bainbridge has discovered a wide range of the Rutt/Etra’s affordances through his live performance visuals and music video productions—including the warped and flowing human figures seen in the 2004 music video for TV On The Radio’s “Staring at the Sun,” which Elliot Jokelson directed (Fig. 7). Bainbridge’s work on this piece not only helped renew interest in analog video processes during the early 2000s but also defined what scan processing looked like for a generation with little exposure to the video art of the 1970s. Further, Bainbridge’s use of the Rutt/Etra informed many software emulations of the synthesizer in the years following.

The first set of Bainbridge’s Rutt/Etra affordances relies upon a circuit that facilitates the use of external modulation signals. Although labeled an “audio input,” this module accepts video signals as well. The input waveforms can then be applied to any parameter in the instrument [33]. Bainbridge’s signature approach passes a parallel copy of the main input video signal through the effects of an MX-30 video mixer before returning it to the Rutt/Etra’s external input as an abstract luma displacement modulation [34].

The second set of Rutt/Etra affordances involves the use of a three-dimensional joystick controller, which Bainbridge compares with Douglas Engelbart’s 1968 premiere of the computer mouse. “You’re really dancing with this joystick” while manipulating the Rutt/Etra parameters in real time and fading between different control voltage sources by hand [35]. The third set of affordances that Bainbridge mentions relies on a slew function, which can be applied in several different ways. Applied to control voltage from the joystick, the slew function smooths out Bainbridge’s manual gestures to provide smoother animations and fine control over image distortions. Applied to the audio input, the slew follows the dynamics of music, and allows the final animation to “bump along with the beat” [36].

In both the research interviews and the community survey, the real-time aspects of historical scan processing instruments are mentioned consistently as among their greatest strengths. The same sources also cite the instruments’ scarcity, fragility, perceived “random” uncontrollability and general lack of state saving and recall systems as their weakest aspects. Such frustrations have inspired several attempts to reenact the Rutt/Etra in particular, within both the analog and digital domains. A reenactment is neither an imitation nor a strict reproduction of an original. Rather, a reenactment of an instrument works through its historical affordances using current technological means to address present contexts. Deeper reflections on the reenactment process can be found in my recent papers for SIGGRAPH [37] and NIME [38].

When Anton Marini began to investigate the Rutt/Etra in 2008, his limited access to the actual instrument itself highlighted the perceived need for a digital emulation. Marini met Bill Etra through a SHARE.nyc event, which Marini describes as an “open jam” for electronic musicians and video artists, and they agreed that Marini might be able to recreate the Rutt/Etra digitally. At the time, Etra was very interested in the idea of a visual orchestra in which many people contributed interactive imagery in real time. Interestingly, Etra was a PC user, and Marini was designing the v002 Rutt/Etra as a Quartz Composer plugin for MacOS. Thus, although Etra could not use Marini’s software, he was deeply interested in any tool that might lead more people to share his artistic vision [39].

In selecting which features of the Rutt/Etra to emulate, and in evaluating whether these emulations were successful, Marini depended largely on Etra’s verbal descriptions together with demonstrations of Bainbridge’s hardware instrument. One hardware feature that Marini was able to observe directly was a military-grade CRT installed in only a few of the synthesizers that Steve Rutt produced [40]. The look and feel of this high-resolution monitor have a quality that is difficult to capture through rescanning and challenging to emulate digitally. On the whole, Marini and Etra’s work brought the pseudo-3D characteristics of luma displacement into a true 3D digital working environment, with full color, lighting effects, and different line types or shading (Fig. 8), while acknowledging that “modern graphics and computer systems make fundamentally different assumptions from analog video systems” [41]. The v002 Rutt/Etra also reenacts Etra’s longstanding interest in low-cost, storable, and recallable digital control over video synthesis parameters [42] through current technological methods.

Marini points out the way that a digital plugin system’s modularity echoes and improves on the original hardware’s modularity, and how he took great pains to understand fundamental analog concepts such as slew, and the way that the high-resolution CRT’s quality far outstripped digital representations of the scan lines [43]. The Quartz Composer framework itself was deprecated in the 2020 Catalina release of MacOS and rendered Marini’s software unusable from that point onward. However, the collaborative aspect of Etra’s vision provided inspiration for the Syphon framework [44], which allows video frames to be shared between MacOS applications in real time.

There are a number of efforts to preserve and provide access to historical scan processing instruments. Notable examples include the work of Dave Sieg in maintaining the last functioning Scanimate systems [45], Daniel Summer’s 2007 “Rutt/Etra Restoration Party” [46], and the residencies for artistic and research exploration of historical hardware offered by Signal Culture [47]. Several digital scan processing emulators also followed Marini’s work. Some, such as the Rutt-Etra-Izer [48] and Inspired by Rutt Etra [49], appear to derive their functionality purely from a survey of existing scan-processed video artworks by the Vasulkas or Bainbridge, for example, or from previous software emulations. Others, like Re:Trace [50], Not-A-Rutt-Etra [51] or my own Vector Synthesis [52], benefit from their authors’ close contact with original historical hardware.

My community survey probed the differences in affordances that the respondents perceived between these software scan processors and the hardware instruments that inspired them. Table 1 presents a short summary of the positive and negative affordances that many respondents noted. Clearly, there are areas where positive and negative overlap. Exciting and unexpected results can tip easily into a feeling of random unpredictability and unreliability when a certain effect is sought or expected. Similarly, a digital system’s precise, repeatable, reliable, and clean output that one artist appreciates can strike another easily as sterile and lifeless. As noted above, the ability to save and recall presets appears frequently as a digital advantage, together with the general accessibility and affordability of software instruments over their hardware counterparts, while hardware’s tactility and aesthetic is considered to be difficult to emulate.

Most relevant to this article are the respondents who had experience with both types of instrument and mentioned noticeable limitations in the digital emulations. These respondents reported that most scan processing software tends to focus on the luma displacement effect, almost to the exclusion of other affordances. Reviewing Marini’s process of reenactment, we see that it combines Etra’s social and artistic contexts with the technical possibilities demonstrated by Bainbridge. This illustrates one potential route to transfer a specific instrument’s affordances to another instrument. However, it also demonstrates how the situational affordances discovered by Sorensen, Hill, or other users had less bearing on Marini’s work and may not be so easily accommodated by his emulation.

Precisely because of the scarcity, fragility, and inaccessibility of scan processing hardware that the survey respondents noted, far more users are likely to encounter one of their digital emulations. The software instruments that follow Marini’s also build upon specific affordances of luma displacement in a 3D environment that are considered the defining characteristic of Rutt/Etra-ness. Other affordances, particularly those related to the use of precision waveforms or the direct input of audio signals to synthesize visual shapes that Sorensen and Hill noted, fall by the wayside. This suggests that the commonly accepted definition of scan processing, and the affordances that one can discover in it, can narrow progressively over time. I find strong parallels between this convergence of affordances and the biases that are affecting artificial intelligence (AI) applications used currently to generate image and sound. While a generative AI system may be trained on an astronomically large number of examples, it is statistically likely to favor features that recur frequently in the training data. Such algorithmic biases have been discussed in the context of large language models [53] and suggest an explanation for the predominance of highly normative aesthetics in AI art [54].

Clearly, the reenactment of historical electronic instruments—whether for sound, moving image, or some combination thereof—is a complex process that maps psychological, social, aesthetic, and technological contexts of the past onto those of the present. One cannot always expect one-to-one correspondences between affordances found in the original and those found in the reenactment. Rather, one should expect new affordances to emerge, given that all affordances are environmental in nature and specific to the situations that give rise to them. My own work constantly revisits primary sources in order to understand their latent possibilities and preserve a sense of connection and continuity with the past. On the other hand, many digital emulation strategies focus on producing simulacra of known results obtained from historical audio/ visual instruments, rather than investigating their affordances. Such approaches risk losing a deeper understanding of the material and social conditions that both gave rise to these tools and shaped their outputs. This makes ensuring functional access to the original instruments—in the studio rather than the museum vitrine—more important than ever.

I thank the following for their time and support: Thomas Fang, Matthew Rempes, Vibeke Sorensen, Gary Hill, Benton C Bainbridge, Elliot Jokelson, Anton Marini, Matthew Schlanger, Victoria Rutt, Signal Culture, Debora and Jason Bernagozzi, Daniel Summer, Paula Petsoulakis, André Holzapfel, Henrik Frisk, and the Swedish Research Council (2019-03694). This article is dedicated to the memory of Louise (Etra) Ledeen, who passed away during its writing.