This paper describes how a culture, a technology, a design process, and a type of image coincided historically and depended on one another to produce their social effects. The conventions of screenshots – photographs of screens – were developed to describe the experience of using computer-aided design on an interactive computer. What was at stake was nothing less than what the word “computer” meant around 1960. While many architects at the time thought of computers as “mere tools,” protagonists of the Computer-Aided Design Project at MIT thought of computers as active partners and simulation environments. Describing screenshots as what Shapin (1984) calls a “literary technology of virtual witnessing” requires explaining how images are used in practice and taking what Latour (2014) derisively calls “iconographic conventions” seriously.
Sometimes in the course of image-making, images are asked to represent unusual things. Around 1960, scientists and engineers working on the Computer-Aided Design (CAD) Project at MIT began imagining that computers could be “active partners” to human designers. They began talking about a future of “human-computer symbiosis.” And they created a new type of image—the screenshot—that represented this new possibility. This paper describes early CAD research as a site for the emergence of the ideal of the interactive computer and how this ideal was described and distributed through screenshots.
Though we now routinely associate computers with interactivity, interactivity was beyond the average user’s experience in 1960. Computers were practically invisible to these users, as well as to the general public. Using a computer typically meant dropping off a stack of punch cards with a technician; a print-out would be waiting to be picked up the next day. For these users, computers were nowhere to be seen, much less interacted with. Because of this, even after interactivity became technically possible, it was nevertheless still possible that nobody would know about it.
Screenshots—photographs of computer screens—were central to the task of constructing a new meaning for the computer. Understanding screenshots requires not only tracking their circulation, but also looking closely at their distinct visual conventions. A key part of this paper, therefore, describes the visual conventions of screenshots that were developed around 1960 and are still in use today.
The episode I discuss is of unusual interest to the history of engineering and design. The Computer-Aided Design Project, which ran from 1959 into the 1960s, represents a revolutionary moment in computation. The protagonists of computer-aided design promoted a concept of the computer that can be characterized by two different notions of interactivity. The first was what Douglas Ross, director of the Computer-Aided Design Project, had in mind when he described the interactive computer as an “active partner” to human designers. Interactivity, for Ross, was modeled on face-to-face dialog. Ivan Sutherland illustrated a second, rather different interpretation of interactivity in his software demonstration, Sketchpad. Similar to the personal computers of today, Sketchpad involved an interactive environment comprised of elements displayed on a screen and manipulated by means of input devices. These two notions combined to constitute a concept of the interactive computer that is still contemporary: the computer as both an intelligent partner and a window onto a mediated environment.
Like the experiments in “simulation” Sherry Turkle (2009) documented at MIT in the early 1980s, computer-aided design around 1960 was a source of discontent. The story of the development of CAD plays out against the background of alternative ways of conceptualizing the computer. One well-known early adopter, the architect Christopher Alexander, summed up the prevailing attitude in a 1964 diatribe against CAD (recounted below). For Alexander, the computer was a tool used only at discrete moments in the design process: when calculation was necessary. At issue in the architectural controversies over the use of computers in the early 1960s was the question of just what kind of medium CAD is and, by extension, what kind of thing an interactive computer is. Is CAD a technology like a slide rule or drafting board? Or is CAD a process, similar to the traditional understanding of the architectural design process? Is a computer a tool to be manipulated or a design partner? On one hand, computers, design, and designers were understood in a way familiar to the profession of architecture. On the other hand, the steps of design and the steps of computer-use were conflated into a single medium, christened “computer-aided design.” The plausibility of both positions was equally founded upon an ambiguity in the concept of a medium: sometimes the word medium refers to a logical process; sometimes it refers to a material technology (Guillory 2010).
Though screenshots represent an important conceptual development, their circulation also had significant practical consequences. Before computer-aided design could be taken up in practice, architects and other professionals needed to know that the interactive computer was a reality. The screenshot began as what Steven Shapin (1984) calls a literary technology of virtual witnessing. Screenshots were used to make a new type of computer-work visible to those without access to the rarefied technology. Though what screenshots represent may now appear self-evident, this is because, to a large extent, we live in the conventional world of interactive computers the protagonists of the Computer-Aided Design Project worked to create and, ultimately, to naturalize.
The next section establishes a theoretical context from the history of computers, visual studies of science, and studies of interactivity. This is followed by an account of the architectural controversies surrounding computation around 1960. The final sections describe the development of computer-aided design, the development of the visual conventions of screenshots, and how and why screenshots were circulated.
2. Concepts of the Computer, Interactivity, and Images
The historiographical importance of taking into consideration what the word “computer” meant in different historical contexts should not be underestimated. Surveying the early years of development of the electronic computer, Michael Mahoney (2005) has noted that “computer” meant something different to each group using one and that many different things called “computers” coexisted. Equating computer with computation, as some historians have done, therefore flattens a rich historical landscape. In a recent essay, for example, Jon Agar (2006) evaluates various claims by historians that certain things in the past would have been “impossible without computers.” He unsurprisingly finds that every computational practice had its pre-computer analogue—computers just made it possible to calculate more quickly. Compiling botanical maps of England with ten-kilometer accuracy or determining molecular structure using x-ray crystallography were possible before computers, even if they were not feasible. Agar’s argument amounts to a tautology: computation is computation, regardless of the technology involved. Agar has elsewhere (2003) argued that the concept of the computer derives from the general-purpose paper-shuffling practices of large government bureaucracy. He does not ask whether the concept of the computer may have changed over time.
Other scholars have, however, asked this question. Peter Galison (1997), for example, describes one conceptual change that went along with the development of computer simulation methods in physics around 1950: “the computer began as a ‘tool’, an object for the manipulation of machines, objects, and equations. But bit by bit…, computer designers deconstructed the notion of a tool itself as the computer came to stand not for a tool, but for nature” (p. 777). This shift from thinking of computers as “tools” to thinking of them as simulation environments had practical consequences. After the shift, virtual experiments could be run in virtual environments, allowing physicists to ask and answer new questions. In other words, a transformation of the concept of the computer (now thought of as a simulation environment) resulted in a transformation of the concept of the experiment (to account for simulation), opening new possibilities. In Agar’s view, everything Galison sees as “new” was possible before the computer (the mathematics of simulation can be computed by hand, after all). In contrast, Galison implies that a concept can be new, and that a new concept can lay the foundation for new practical possibilities.
The concept that Galison describes as “simulation” could also be described as “interactivity.” More precisely, simulation is one of two distinct types of interactivity that emerged around 1960. In the first, older definition, interaction means a type of communication modeled on human-to-human dialog (see, e.g., Rafaeli 1988; Steuer 1992). The key feature of this definition is that, as in a dialog between people, previous exchanges are taken into account in any current exchange; this requires memory and the processing capabilities to make use of it. Few computers in 1960 were described as being interactive in this way. A second, more recent definition of interactivity requires the creation of an environment with which to interact (see, e.g., Steuer 1992; De Vos 2000). In this view, the interactive computer provides a virtual world in which the user can create and manipulate virtual objects. While the first definition of interactivity relies on a verbal exchange, the second definition relies on a sense of “direct manipulation.”
A crucial difference between these two definitions is the level of abstraction at which they operate. A dialog can be carried out through speech, an exchange of notes on paper, or a back-and-forth process of input and output using a keyboard and terminal. To a large extent, the medium of exchange can be abstracted away. Environmental interactivity, in contrast, depends on a sense of presence created through the use of the input and output devices themselves. For this reason, environmental interactivity requires “peripheral devices” of a certain type: to begin with, they need to be low-latency and they must map a diverse range of possible actions onto factors in the mediated environment (Steuer 1992). It seems, therefore, that the concept of environmental interactivity was modeled on the real-time manipulation of objects in a virtual space through the portal of a screen; that is, the metaphor of interactivity seems to have been borrowed from the experience of using computer-aided design systems.
Because interactive computers were rare in 1960, the general public was likely to hear about them through the popular scientific press. J. C. R. Licklider’s 1960 article describing a future of “man-computer symbiosis,” for example, likely prompted readers to ponder what it would be like to interact with an interactive computer. The answer came, in part, through images. The first screenshots were created not only to describe what the interactive computer and CAD can do, but also to represent the fact that they were not merely possibilities, but a reality. A change in representational practices often parallels a change in the common perception of reality. The spread of steam-powered locomotives across the nineteenth-century countryside, for example, went along with new depictions of the American landscape and changing ideas about the relationship between humans and nature (Marx 1964). New landscape paintings – with trains – described something new in the world, and the museum-going public who contemplated them came to understand the world differently. Likewise, new visual conventions were honed to represent the new interactive computer. I call images following these conventions “screenshots,” a usage which is admittedly somewhat anachronistic. The Oxford English Dictionary records the first use of the term in 1983. The word has the same root as “snapshot,” and its basic meaning is similar: a screenshot is a relatively informal photograph of a screen. Computer screens were uncommon before 1960, so it is no surprise that screenshots were uncommon as well, and that it took some time before they were recognized with a name of their own.
To understand screenshots and the interactive computer, we need to look closely at the images and at their circulation. Doing both of these things is still somewhat unusual in discussions of scientific images. One analytical pattern has been to focus on how images are used, not on the specifics of how they show what they show. An example appears in a seminal collection of essays, Representation in Scientific Practice (Lynch and Woolgar 1990): Bruno Latour ( 1990) describes, among other things, how a map of an island was recorded on paper and transported around the globe, and how its importance and its power derived from its immutability and its mobility. In Latour’s telling, there is no need to describe the map—his argument does not rely on what the map showed or how it showed it. Latour (2014) more recently summed up the now-conventional wisdom he helped create: “One should not isolate the scientific imagery and shoehorn it into the types of questions raised by iconography. There is nothing visual in scientific visual imagery. Literally, there is nothing to be ‘seen’” (p. 349). In this way of thinking, an image is nothing more than a link in a “referential chain”—it has no meaning outside of its circumscribed context. The action is not in the image, but in the networks in which it takes place. The spread of this analytical trend seems, if anything, to have widened. The editors of a 2014 follow-up to the 1990 volume (Coopmans et al.), for example, note that, in recent discussions of scientific representation, the “turn to practice” is taken for granted in a way it was not when the initial volume was collected.
If images are seen as playing supporting roles in scientific work, the danger is that what they show will be taken as a foregone conclusion. The alternative is to take iconographic conventions seriously. This approach has its adherents. Also in Representation in Scientific Practice, for example, John Law and Michael Lynch (1990) take a close look at how birds are represented in various field guides. Choices of line weight and shading, using photographs or drawings, the inclusion or not of circumstantial details, and the positions in which birds are shown all affect the knowledge the images produce.
When discussions of practice are combined with visual analysis, the two often compete for attention. Shapin’s (1984) well-known discussion of literary technologies of virtual witnessing is a case in point. Though he briefly describes the visual conventions of Robert Boyle’s drawings of air pumps (focusing in particular on the convention of including a large amount of circumstantial detail to heighten realism), Shapin moves quickly to describe how this technique was used by Boyle to convince his readers. In the rush to discuss the use of images, some of the analytical possibilities the images present are missed. Lorraine Daston’s (1988) essay on the “factual sensibility,” for example, describes how the visual juxtaposition of objects in cabinets of curiosity generated an appreciation of “facts” as individual, specific things. Looking again at Boyle’s drawings, then, we can perhaps see juxtaposition being used to a similar effect. That is to say, the character of the reality Shapin describes has irreducibly visual features that could likely be described with greater specificity to produce further analytical results. It seems that Shapin does not dig further into details of “iconography” such as these because it would have competed with his central discussion of how images were used.
The account below aims to account for both the specific visual conventions of screenshots and how and why they were used. But before describing the new reality of the interactive computer and the screenshots that would promote it, it is worth recalling what “computer” meant previously.
3. Computers and Architects circa 1960
For most people in 1960, a computer was something with a very limited function, operated by technicians and used only at a distance in a rather abstract way. As memorably summed up in the title of a recent book, When Computers Were Human (Grier 2007), a computer was typically seen as equivalent to a room full of people crunching tables of numbers. Both calculated. A fast computer in 1960 would have done about 100,000 calculations per second, and this calculation time would be divided up and divvied out to different computational tasks. Someone with access to a computer who wanted to run a program would drop off a stack of punch cards and come back the next day to pick up the print-out. Using a computer was conceptually no different from setting a room full of people to work on a calculation. Computers, like human calculators, were hidden inside the black box of an abstract function.
This way of conceptualizing the computer had practical consequences. Computers were often assigned a lowly status in the practice of various professions, following divisions of labor that were established long before computers entered the scene. Jay Wickersham (2010) describes how the first large architecture firms in late-nineteenth century Chicago (such as Adler & Sullivan and Burnham & Root) began employing “mass-production principles of specialized labor:”
Burnham understood that the key to successful large-scale practice was to rely on the skills of colleagues and assistants. “The only way to handle a big business is to delegate, delegate, delegate!” he once snapped at his partner, when he saw Root getting bogged down in routine work. (Wickersham 2010, p. 22)
Bracketing, obscuring, or black-boxing people and their labor is a familiar theme. Shapin moved on from his study of Boyle’s literary technologies (1984) mentioned above to investigate the invisible technicians in Boyle’s laboratory (1989). He found dozens of “laborants, operators, artificers, and servants” (1989, p. 555) who were obscured by Boyle himself (who rarely mentioned them) and by later historians of science (who preferred to portray science as a “formal and wholly rational enterprise carried out by reflective individual thinkers” [1989, p. 563]). Computers in 1960 were also doubly invisible in architecture, but in a different way. An engineer’s assistant or a draftsman, far removed from the type of work that warranted partnership in a firm, would offload his own tasks to the black box of the computer, if he was lucky enough to have one available. Rather than laboriously calculating the stresses on dozens of structural members he could create a computer program on a set of punch cards, then hand it off to be calculated. The results would return to him to be deciphered, and he would move his findings back up the chain of decision making. Not only was the computer barely present to those who used it, but the user himself was invisible within the design process.
This suited the profession of architecture just fine. Mario Carpo (2011) has explained how, starting in the fifteenth century, architecture came to be associated with an abstract quality—design—rather than with buildings and their construction or even drawings and their details. Calculation hardly registers in this moral economy—the value lies elsewhere.1
In this context, the interactive computer was not merely a distraction or a curiosity (let alone “just a tool”), but something at odds with the established order. The very idea of computer-aided design was attacked with a passion reserved for false idols. Architects weighed in on the subject well before computer use was common. In a critique of 1964 like many others, Christopher Alexander, a famous early adopter of computers, pulled no punches in a speech delivered at Architecture and the Computer, the first conference on computer-aided design:
Anybody who asks “how can we apply the computer to architecture?” is dangerous, naive, and foolish. He is foolish because only a foolish person wants to use a tool before he has a reason for needing it. He is naive because, as the thousand clerks have shown us [in other words, the fact that human calculators and computers are essentially the same thing], there is really very little a computer can do. And he is dangerous because his preoccupation may actually prevent us from… seeing problems as they really are. (Alexander 1964, p. 54)
Alexander’s representational habits match his pronouncements. Alise Upitis (2013) has shown that Alexander’s way of thinking about computers fit the IBM mainframes he used between 1961 and 1963. These computers were far from interactive. Alexander would create programs on punch cards and hand them to a technician through a window, and he would pick up a table of results the next day (Fig. 1). These he would re-interpret though elaborate hand-drawn diagrams (Fig. 2), which would in turn take their limited place in his design process. In these two images we are presented with a compelling picture of how an architect and a computer can work together. It is a tidy sequence, codified in professional practice textbooks: design begins as a hunch or a sketch, proceeds through a phase of iterative development, and results in a set of definitive drawings and specifications. In the middle phase, the architect works with information from a variety of sources; for Alexander, the computer’s calculation was one such source. The computer itself, set up to play this role, is just as tidy: a carefully encoded program is taken as input, the program is processed, and output is handed back for decoding and interpretation. Alexander’s practice makes sense as a “chain of transformations,” and, on one level (and to agree with Latour’s (2014) analysis, described above) it would not make much sense to look at his images outside of the design process in which they were used. There is “nothing to be seen” in the printout (it was meant to be transformed into a diagram), and there is not much to be seen in the diagram (it is part of a set of instructions on how to design a building). Other images with different visual conventions would have achieved the same effect in Alexander’s design process.
Alexander was, however, keenly aware of the connotations of his images, and he was very successful at using them for self-promotion. We would therefore be justified in examining his images outside of the process of which they were a part. The branching diagram connotes, among other things, the idea of rational, authorial synthesis; the printout connotes the idea of an objective, rule-based design process. Because the screenshots of CAD presented below were likewise created for “public relations purposes,” it will be important to examine connotations such as these — to see the images not only within the process to which they belong, but also in isolation, as representations of the process itself.
4. Designing Computer-Aided Design
Computer-aided design and the interactive computer needed a public-relations campaign in 1960 because their novelty was easy to miss. No single event marked their arrival. The prerequisites of interactivity included a programmable computer with a real-time display, suitable input devices, and sophisticated software. Certain computers had met these requirements for nearly a decade, but it would be a stretch to describe these earlier computers as fully interactive in the sense given above. The Whirlwind computer, for example, which was operational in 1951, was famously the “first real-time digital computer” with “the first practical use of an oscilloscope or CRT as a graphical output device” (Weisberg 2008). The first computer animation (a bouncing ball) was programmed on its 64 by 64 pixel display, as was the first video game (trying to get a ball to go through a hole by changing the frequency used in a calculation). Impressive though all this was, Whirlwind’s programs had the character of tricks or hacks rather than a new reality to be shared.2
Setting this closed-minded ethos aside, if there was one concrete feature missing in earlier systems such as Whirlwind, it was the type of rapid dialog that would allow the computer to appear to be an intelligent partner to its operator. This conceptual ideal was what Licklider described as “man-computer symbiosis.” Licklider wrote eloquently of a future in which “human brains and computing machines will be coupled together very tightly,” and of the “resulting partnership” that “will think as no human brain has ever thought” (1960, p. 4).
In his statement of objectives for the Computer-Aided Design Project, Ross (1960) characterized the goals for CAD in terms similar to those of Licklider: a CAD system would be an “active partner” to designers “throughout the design process” (Ross 1960, p. v). Ross stressed that CAD would not be a “mere tool” to solve problems, but a partner in figuring out how to formulate a problem in the first place. CAD would not only help with drafting and calculating, but also “accept and analyze… sketches” using “the entire sweep of the scientific method” (1960, p. 15). Ross described how this would work in terms compatible with what I have called environmental interactivity: “the user must be able to establish a controlled environment, set up an experiment, try test cases, analyze results and modify whatever is appropriate, all by simulation on the computer” (1960, p. 15). Ross’s overriding metaphor, however, was not simulation but dialog between human partners: “all of these various facets… must be carried out efficiently and conveniently by statements back and forth in the language, in a conversation or discussion about the problem, between the man and the computer” (1960, p. 15). Computer-aided design was modeled, for Ross, on communication—not face-to-face, but face-to-interface.
So the Computer-Aided Design project followed closely on the technical developments of Whirlwind and the conceptual developments of Licklider. The novelty was in its scope, not its particulars. The Computer-Aided Design Project began as a generalization of Ross’s earlier project to create a language for the control of fabrication equipment (Weisberg 2008). Ross’s goal was to create a fully computer-based system that could be used to design and eventually manufacture a wide range of physical objects, from airplanes to bridges. At the beginning of the Computer-Aided Design Project, a five-year-old system (neither cutting-edge nor antiquated), the TX-0, was transferred from the Lincoln Laboratory to MIT for use by CAD researchers. Two other protagonists of the Computer-Aided Design Project, Robert Mann and Steven Coons, compiled a list of areas of investigation that would be brought under umbrella of CAD research:
A graphical input device…
A graphical output device…
A symbolic input device…
Symbolic output devices…
A translating system for converting designer’s language to machine language and converse;
A shape description memory system;
A shape description computation system…
Programs for strength… calculations…
A catalog of standard parts… (Coons and Mann 1960, p. 11)
The crowning achievement of the Computer-Aided Design Project was also one of the most iconic software demonstrations of the era: Ivan Sutherland’s Sketchpad, created in 1963. Sketchpad represented a vision for CAD that has not been superseded. The Sketchpad operator could draw directly on a cathode ray tube with a light pen, using a keyboard and panel of buttons to trigger various actions. One click would position the endpoint of a line; another click would locate its other end. Geometrical entities drawn in this way could be changed or moved around, or even copied, resized, and nested within one another. Annotations and calculations took place within the same interface. After sketching a truss bridge, a routine could be called up to calculate the stress on its members (Fig. 3). The movements of virtual mechanical assemblies could be visualized with a different routine (Fig. 4). In Sketchpad III (the three-dimensional version of the software), plans, elevations, and views could be generated from a virtual model (Fig. 5). In the end, drawings and specifications could be plotted. At any point within the design process, earlier phases could be seamlessly revisited: if stress calculations revealed problems, the original drawing could be altered by resizing members, moving or adding pieces, and so on. The “final” drawings would be automatically updated and ready to plot again.
With the computer now acting as a partner, it would not be easy to protect the professional judgment of the architect from encroachment. The neat division of labor Alexander had argued for—in which “there is very little a computer can do”—would no longer be sustainable. Coons’ presentation at the Architecture and the Computer conference, at which Alexander also presented, offers one example of this line of argument:
… many architects, and the archetypal members of their coterie, artists, use the word “design” to mean only the innovative, generative, intuitive acts of conception; to them, the necessities of structural analysis, the mechanics of heat flow, the aerodynamics of wind loading, and other analytical methods for coping with the stringencies of nature and natural law, while recognized as essential processes, are not considered a proper part of design, but only of engineering and construction; subservient functions that must be employed to bring a concept to realization. … The true and complete process of design, it seems to me, consists of an inextricable mixture of these intuitive, imaginative cognitive processes together with analytical, mathematical, rational processes. (Coons 1964, p. 26)
Coons goes on to describe “an architect (or an engineer) seated at a computer console of the future” (Coons 1964, p. 26). During the computer-aided design process, tasks of various kinds alternate freely between the computer and its operator. At one point Coons suggests that the computer can displace human judgment: when trying to resolve an incompatible set of constraints, the operator can “leave the entire problem of adjustment to the computer, content to accept whatever result it achieves.” Doing so would mean accepting that the computer is “in a certain sense intelligent.” Computer-aided design would not only combine the strengths of the computer and the designer, but conflate what Alexander saw as their essential characteristics: the computer would be creative and the designer would be analytical.
Coons illustrates his presentation with screenshots from Sketchpad (Fig. 6). For Coons—and eventually many other people—Sketchpad represented the point at which the interactive computer congealed into a conceptual ideal. The term “computer” has since come to be redefined around this new model. Using older, pre-interactive computers, users would design a program, encode it in punch cards, and drop it off; the program would be run, results printed, and the printout interpreted. Each was a discrete phase. With the interactive computer, design and computation no longer took place in their own time and place, with their own tools: both occurred in the same virtual space “within” the screen, with the human and the computer sharing in the action.
Sketchpad placed the protagonists of CAD in an awkward position. If CAD was a success, the danger was that it would recede into the background as a “natural” part of the design process. Ross (1963) said that his desire for CAD was to create a system that would allow designers to “think almost entirely at the concept level within [their] own field of interest, while at the same time carrying out data processing activities of extreme complexity” (p. 305). In other words, he hoped for a contradiction: that CAD would be a medium that allowed immediacy.
Guillory (2010) has noted that the dream of immediacy is a recurring theme in the history of communication, and that it comes along with two conflicting ideas of what a medium is. A medium, Guillory notes, can be either a process (such as painting or design) or the material technology through which a process is carried out (such as paint and canvas or the interactive computer system). To use Guillory’s historical example: John Locke thought of words as a medium for thought while his contemporary, John Wilkins, thought of writing as a medium for speech. The concept at work in each case is distinct. In Guillory’s characterization, writing involves ink and paper and is meant to bridge a physical gap between people; it is a technology, something external. Words, on the other hand, are cognitive phenomena synonymous with the interior thought process. The important point here is that the medium concept itself is ambiguous: these two meanings can be distinguished, but not cleanly or completely. A physical medium such as writing must of course be used to communicate ideas, and thought required the long physical process of enculturation.
One practical consequence of the ambiguity in the concept of a medium is that, historically, what people have thought about technical innovations has often influenced their notions of how thought operates. Some seventeenth-century thinkers believed, according to Guillory, that a technical innovation might change thought itself, as in the Enlightenment dream of a future of perfect communication modeled on printing. The very concept of reasoned communication, in this case, came in part from the properties of a technical medium.
The ambitions for computer-aided design relied on a similar conflation between the properties of a material technology (the computer as a “tool”) and notions of how cognition or problem-solving work (i.e., through dialog or through the manipulation of objects in a virtual environment). Given the conceptual ambiguity involved, it should perhaps be no surprise that yet another intermediary—an image type, the screenshot—was developed to aid explanation.
5. Representing CAD and the Interactive Computer
As a matter of historical record, the first screenshots were used to explain the first interactive computers. It has been claimed that the very first screenshot was a photograph of a cathode ray tube displaying a pin-up girl, taken in 1959 (Fig. 7; Edwards 2013a). The system on which it was displayed was among the first interactive computers, one of the SAGE systems developed in the late 1950s for air defense. Technicians programmed the computer to display the drawing for diagnostic purposes. One of them took a photograph.
Most screenshots circa 1960 were somewhat more formal. Sophisticated systems often had a special apparatus that kept the camera in the right place in front of the screen (Fig. 8). Computers sometimes had two identical screens displaying the same image, one of which would be used interactively by its operator while the other had a camera mounted to it. To accurately record what was being displayed on the screens, all that had to be done was to push a button to operate the shutter.
Screenshots are, however, more than mere photographs: they employ a distinct set of conventions. These conventions center on the fact that screenshots are, indeed, photographs of interactive screens. That is, the fact that they are photographs of screens is part of their representational content. So figure 7 is not simply an image of a pin-up girl, but an image of a pin-up girl displayed on an interactive computer screen.
Before the conventions of the screenshot were established, photographs of screens were not clearly distinguished as a medium. In 1960 there were several other ways to output an image from a computer. Images could be printed or plotted as easily as they could be displayed on a cathode ray tube. Screen-mounted cameras were first used as output devices to produce drawings which were conceptually no different from the era’s prints and plots. Presenting the output options available in 1964 to his peers, a technician at IBM listed “cathode ray tube drawings” next to “drafting machine drawings” and “X Y plotter drawings” (Smith 1964, p. 57). Each had its own features and limitations, but all were basically interchangeable; a screenshot was simply the quickest option in some cases: “The primary characteristic of the Cathode Ray Tube as a graphic device is speed. Drawings that might take 15 minutes to an hour on a drafting machine or x y plotter can be produced in seconds on a C. R. T.” (Smith 1964, p. 57). The differences between the example CRT drawings he shows (e.g., Fig. 9) and drawings produced other ways are almost imperceptible.
Coding an image as a “photograph of a screen” rather than a “drawing” requires that deliberate choices be made. Screenshots differed from the IBM technician’s cathode ray tube drawings significantly. A white or monotone drawing on a black background, for example, signifies the dark screen lit up by an electron gun. (In CRT drawings, on the other hand, black and white are inverted so that they look like normal ink-on-paper drawings.)
Conventions such as having a black background were settled through experimentation. In some cases, inverted and non-inverted versions of the same image were published in different venues, but for no discernible reason. Figures 10 and 11, for example, show how Sketchpad displays numbers; one is a CRT drawing with a white background, the other a screenshot with a black background. Their content is largely the same, but the screenshot conveys an extra bit of information: the fact that the image was captured directly from a screen. The extra semiotic content may be unimportant in this case, but sometimes—such as when the content of the image is little more than “something exciting we can do with an interactive screen”—the sense of it being “from the screen” was the most important content of the image.
Some things computers could do in 1960 eluded the other image types that were available. How, for example, could one capture the “twinkle” of a cathode ray tube? As the caption for one of the figures (Fig. 12) in the technical report describing Sketchpad explains, this was indeed a matter of concern:
Displaying the spots of a large display in random sequence makes the display appear to “twinkle.” This photograph was exposed only long enough to show about half of the spots of a twinkling display. It conveys the impression of a twinkling display as well as any still picture can. (Sutherland  2003, p. 65)
In the caption for his twinkling screenshot, Sutherland mentions that he wants to convey the “impression” of a screen. This is a type of representational problem that helps drive the development of artistic media. The development of Neo-Impressionist painting in the 1870s, for example, has been descried as the result of experiments in representing optical phenomena that previous painting techniques failed to represent (e.g., Foa 2015). Trial and error added new conventions to the repertoire.
showing examples of what software can do rather than a single definitive image of a project (fig. 6). This emphasizes that the image on the screen is easily alterable, thus implying that the screen from which it came is interactive.
showing incomplete or partial views, which emphasizes that the computer screen offers a framed view of a virtual object with a reality beyond any particular representation (fig. 13).
implying that what is shown in the image involves computation in some way (fig. 3). In the case of computer-aided design, this meant showing annotations, end points of lines, and so on. These details suggest that computer-aided design combines the virtues of sketching with the virtues of math and logic. The impression is that the drawing is under strict computational control, but still subject to quick alteration by its user.
a look that is, by the standards of other media, unpolished and without the normal niceties of visual communication (fig. 14). The pixelated, jagged lines and lack of context of screenshots convey the impression that a practical task is underway on a device with limited graphic ability (such as a cathode ray tube).
These conventions add up to an image type that represents the interactive computer and the process of using one. They show not only particular things (this or that drawing of a geometrical object), but, more importantly, they give a second-hand impression of the experience of using an interactive computer. In his discussion of the images Boyle used to create a “virtual witnessing” experience, Shapin (1984) identifies one crucial convention: the inclusion in an image of a great “density of circumstantial detail” (1984, p. 481). This convention is also crucial for screenshots; perhaps all the conventions listed above capture different circumstantial details. These details create, in the viewer’s mind, an effect of realism. In the case of screenshots of CAD in action, what is real is not any particular design, but the virtual environments and human-computer symbiosis that constitute the interactive computer.
6. Circulating Screenshots
In order for screenshots to convey the reality of the interactive computer, they had to circulate. Under normal circumstances, screenshots were as ephemeral as the fleeting images they captured. In most professional workflows (such as that of an architect) incidental sketches and printouts are rarely kept. Figure 15 shows a typical sequence of images. The top image is a difficult-to-decipher, computer-created plot of an optimal flight path through a mountain range; the bottom image is an interpretation produced by someone familiar with the topography and the software used. The traces of the computer that can be seen in the first image are erased in the latter. The only thing unusual about these images is that the computer-generated drawing was not thrown away. Why it was kept is telling: this sequence of images were presented by a pioneer in the professional use of computers for graphic purposes, W. A. Fetter, at the same conference in which Alexander delivered the diatribe and Coons the rebuttal cited above. Fetter’s images were used precisely to make the interactive computer present in a discussion about its worth.
The task for which screenshots were used by Fetter, Coons, Sutherland, and others was to convince people to see computers differently and to change their professional habits of computer use. Computer-aided design and interactive computers were new technologies with which few had experience.3 Sketchpad itself was designed to run on a one-of-a-kind computer, the TX-2, at the restricted-access Lincoln Laboratory in Bedford, Massachusetts. For a project about interactivity, such inaccessibility was a serious problem.
Screenshots seem to have reached only small groups in 1960, but they extended their reach to ever-larger audiences in the first years of the decade. The first screenshots that circulated outside of small circles of colleagues were presented at conferences using slide projectors (Fig. 8). Screenshots showed up at the Joint Computer Conferences starting in about 1962, and they made several appearances at the 1964 SHARE Design Automation Workshop. Soon after, screenshots were presented to wider professional audiences. In 1964 screenshots were shown to architects at the Architecture and the Computer conference by Fetter and Coons. Trade journals followed conferences, expanding the reach of screenshots further. Screenshots of Sketchpad were first seen at a SHARE conference (in 1963), later in the trade journal SIMULATION (in 1964), and by 1966 they were featured on the pages of Scientific American and Design Quarterly in popular articles by Sutherland and Coons.
It is worth noting that the success of Sketchpad was an anomaly. Lincoln Laboratory did not generally succeed in showing the world the projects it was working on. According to the reminiscences of those who were there, “much, if not most, of the work there has slipped from our collective consciousness, with Sutherland’s ‘Sketchpad’ system being the notable exception”—and this despite “almost all” the projects being “worthy of our attention” (Buxton 2005, p. 1162). Lincoln Laboratory was famous for developing expensive and conceptually avant-garde technical projects into mock-ups that were exhibited in one-on-one demonstrations. Sketchpad was one such mock-up, and although Sutherland intended Sketchpad to be developed further, it was not used outside academia. Commercial CAD systems only arrived later in the decade (see Kemper 1985). The success of Sketchpad had to do with how photogenic it was. While Sketchpad and the unique computer it was programmed for became obsolete and disappeared, the screenshots live on.
By 1970, screenshots were common, and they remain so today. As a genre they have had an influence of their own. Robert Bruegmann (1989) describes how computational aesthetics (promulgated in part through screenshots) entered architectural culture: the Centre Pompidou (1970) by Renzo Piano and Richard Rogers, for example, combined a “logical” organization with a wireframe-graphics sensibility.
Much had changed by this point. The hardware and software needed for CAD had been neatly packaged, and the stage had been set for the confrontation between traditional design sensibilities and “simulation” that Turkle (2009) saw playing out in the early 1980s across MIT’s school of architecture. Interactivity was no longer new and rare, but something to be discovered by an entire generation of architecture students—in short, it was a common point of cultural reference. The ensuing battle over computer-use centered on representational practices. Turkle describes how older professors rejected printouts and clung to their (non-electronic) sketchpads, a medium they thought generated a closer, more personal relationship with the design process. But everyone seems to have agreed that computer-use was inevitable. As a compromise with the future, students began to experiment with “softening” computer printouts by drawing over them with colored pencils. Designing on a computer was seen as having certain advantages, but computer-use was not thought of as offering the symbiotic, immediate relationship that Ross and Sutherland had hoped for.
The practical success of the interactive computer had led, perhaps, to a waning of the ideal of interactivity. The students Turkle describes carried out their design process on interactive computers, but this was a fact they were not interested in publicizing, so they did not use screenshots, but rather printouts of normal, finalized architectural drawings—and along with these other representational practices came other values and ideals. Screenshots take time and effort to produce and circulate (although less today than in 1960); they therefore carry the mark of a circumscribed conceptual world.
7. Conclusion: Productive Conflations
The interactive computer became a possibility in the late 1950s, and computer-aided design was among the first demonstrations of such a system. Even after it was possible, however, the new conceptual ideal of the interactive computer was not widely known. The first screenshots bridged this gap; they were used to describe the interactive computer to people who had never used one. Sketchpad, the first demonstration of CAD, was the subject of many early screenshots, and it became an early exemplar of interactivity. More than this, Sketchpad came to stand in for the entire culture of early-1960s experimentation around the interactive computer.
Prior to the personal computer revolution, when people talked about CAD or interactive computers, they were usually talking about static images – that is, discussion focused on something far from interactive, and far from the computers themselves. Because screenshots came to stand in, at a distance, for so many things (a design process, a material technology, a culture), the properties of screenshots and the properties of these things began to blend together. The resulting conflation of properties produced some notable results. First, confusion occurred between what was shown in a screenshot and what was possible to do with a computer. Because they are understood as straightforward representations of realty (simple copies of images on screens), screenshots serve as excellent, though sometimes misleading, pieces of evidence. Once mobilized in a screenshot and taken far enough away from the computer that a live demonstration is impossible, there is little choice but to take the “reality” a screenshot conveys at face value. The resultant ambiguity between being able to do something (now and in the future) and actually having done it (often only once in the past) became central to the culture surrounding the computers circa 1960. Computer scientists such as Ross and Sutherland came to value systems that could solve any problem more highly than any particular solution itself. Software like Sketchpad may have only run a few times without being used to solve any real problem, but the screenshots remain, and with them the plausibility of its grandiose ambitions.
Computer scientists used the ambiguity between the possibility of being able to do something and the practicality of doing it to their advantage. At the end of his 1963 technical report on Sketchpad, for example, Sutherland illustrates some of the potential uses of the program. He shows how an operator can make patterns, mechanical linkages, artistic drawings, stress diagrams of bridges, and so on. All this looks great—and in fact some of it is too good to be true. Sutherland helpfully follows his examples with a section comparing their cost (in terms of how many hours it took to set up the demonstration) with the cost of conventional alternatives. Not all compare favorably. But these caveats did not dampen the success of Sketchpad; Sutherland argued that they were a good reason to keep working on computer-aided design, to make it better. With the help of screenshots, abstract possibility had already been transposed into reality; all that was left was to work out the details.
A second result of the conflation of screenshots with CAD was that something exclusive and intimate—the rare interactive computer—could be made present in the public realm. This was not inevitable or obvious. Practitioners such as Alexander insisted that computers be kept at a distance from design precisely because the latter was of such great social importance. In arguing for intellectual and physical closeness with computers, proponents of interactivity risked creating a situation in which design would retreat to the closed world of those huddled around computers. Screenshots resolved this dilemma by conveying a feeling of closeness with computers, but at a distance, through the intermediary of printed paper or projected image.
This counterintuitive effect of screenshots had a further result: the promise of computer-aided design, which has to do with closeness and interactivity, needed to be taken on faith. A piece of paper is obviously not interactive, but accepting a screenshot means believing that interactivity exists. Goodwin (1994) has argued that seeing images in specialized ways (and asking others to see them in the same way) is a foundation of professional expertise. Screenshots therefore publicizes the specific expertise of the CAD operator: they creates the impression that someone, somewhere is in control of the interactive computer.
These conflations had the final result of transforming computer-aided design into a conceptual ideal worthy of being a mission. Once screenshots allowed the experience of using an interactive computer to be described and distributed, the effects of CAD reached beyond any particular design solution or narrow user group to design and society more generally. Screenshots continue to serve as the foundation of the ongoing conversation about the reality (and the potential) of the interactive computer.
Though Carpo’s account matches the contemporary, mainstream view of architecture, it misses the practices (and there were many) that combined ideals that would later be divided between architecture and engineering. One example is the controversy surrounding freestanding columns in eighteenth-century religious architecture; a large contingent of architects at the time sought to register the “circulation of forces” through the articulation of buildings into parts (Picon 2004). In other words, it would be difficult in this case (and others) to dissociate “design” from what could be called “engineering ideals.”
The culture of computing in the 1950s has been characterized as a “black art” practiced by a “priesthood” jealous of their intricate knowledge of one-of-a-kind systems (Backus 1980).
By far largest group who had experience with an interactive computer by 1960 was the technicians and users working on and with the Semi-Automatic Ground Environment (SAGE) systems developed by the US military starting in about 1955. SAGE was a monumental effort that required the training of nearly 8,000 programmers by 1960, who later were dispersed throughout the emerging software industry (Campbell-Kelly 2003, p. 40).