All Watched Over by Our Data Double

In 1967, the American author Richard Brautigan wrote a prescient poem on the nature of the then-emergent technology of computing and its subsequent evolution into information and communications technology and social media.

In his poem, Brautigan has anticipated the primary function that computers and their networks would assume in the post-Internet age: a pervasive, distributed, and totalizing social surveillance system that de-personalizes and objectifies the subject (the user), as a data construct. The subject is thus objectified as a data double, a representation of itself within an all-encompassing algorithmic system.

The data double [1] is a term used to describe the representations and informational flows [2] generated by the users of networked communications technology to allow the agencies (corporate; governmental; and supra-governmental) managing those systems to construct and monitor models of its users. Through this panoptic process, the user (subject) is observed (browsing, liking, retweeting, etc.) and thus objectified. The modeling system’s algorithms track not only individual users but also other users’ interactions with them, each user contributing to the construction of each other user’s data double. Participation in this surveillance-panoptical system is inescapable, the subject confined not by one invisible guard but as many guards as there are inmates (some more in/visible than others): “To create a ‘society of control’ where people would be watched all the time . . . the idea was that discipline would be internalised and the need for the inspector, the watching itself, would be eventually exhausted” [3].

The capture and gathering of data from human activity has been described by Philip E. Agre as two cultural models of privacy [4]. The capture model structures technological processes through linguistic metaphors that are closely associated with information technologies, real-time processes, and industrial workflows. The more dominant and historically precedent surveillance model structures data collection through visual metaphors that are often of a political character, such as the panopticon.

Here, we consider the panopticon as a surveillance metaphor and contemplate more recently developed propositions, such as the synopticon [5] and Fernanda Bruno’s concept of “lateral surveillance” [6], that account for the distributed nature of contemporary surveillance instruments. This inverts the panoptic algorithm as surveillance by the many of the few. Rodney Jones has observed the shifting characteristics of the panopticon:

Jones notes the merging of information technology with the material infrastructure of the modern urban and domestic environment, but the advent of big data and surveillance capitalism [8] supercharges this scenario. Benjamin H. Bratton’s Stack model [9] proposes a complex network based on technological ubiquity. The Stack proposes the emergence of an “accidental megastructure” (2015) of computing and geopolitics where we no longer exist as citizens but only as users.

Within this context, we can now consider ourselves constructed as a double; an object we would be unlikely to recognize but that remains a substantive representation of ourselves in the algorithmic ecology the data double inhabits. Information and communication technologies can be envisaged as a mirror maze of pervasive and all-encompassing surveillance, a parallel reality or model generated by the tracking of user behavior and inhabited by users’ data doubles.

Göran Bolin and Anne Jerslev argue the data double should be considered less a representation than a correlate: “The correlational and non-representational structure of big data analysis is highly abstract and difficult to fathom . . . . ‘Non-representational’ simply means that there are no real referents in social reality since it is the correlation that is the point of data analysis” [10].

A correlate can be understood as a complement or supplement of something else. In the domain of big data, a correlate is a statistical relationship between two variables. A relevant example might be found in the similarities between consumer behavior (as modeled by networked surveillance systems) and user preferences (e.g. patterns of purchasing behavior). In the network surveillance industry, the process of analyzing such consumer behavior is known as sentiment analysis [11] and often involves the application of machine learning to text analysis, natural language processing, pattern recognition, and—increasingly—biometrics (e.g. facial-, pose-, and affect-recognition systems).

Tus, the data double might be considered an abstraction of the human subject, aggregated, analyzed, and mined for its similarities and dissimilarities with other data doubles: a pattern, like a fingerprint or QR code, within a pattern recognition system. The data double might be imagined as layers of waves or a grating that forms part of a massively multi-layered apparatus of patterns resembling an interferogram. This could be envisaged as a holographic space of correlates (data doubles) manifesting as shimmering moiré patterns of data, their shifting, wavelike forms superimposed to create emergent, iridescent structures.

It is difficult to imagine such correlates resembling our physical selves. However, it is likely that if we could observe this shadow-world of aggregated data doubles, we might recognize our traces, perhaps as ripples of agency in the shimmering patterns, in a manner similar to how we recognize others from the smallest signature element—the profile of a nose or a quality of movement in a crowd, or an accent amongst many voices. Humans are expert pattern (re)cognizers who, like the machine-learning algorithms that underpin networked surveillance, are able to generate connections and predictions.

Shoshana Zuboff has observed that the primary product emerging from the excess value generated by the big data-driven collection and aggregation of data doubles is the capability of predicting the future behavior of individuals and populations. Zuboff argues that prediction is the primary currency of what she designates the “age of surveillance capitalism” [12]. Being able to predict the future can be profitable.

It has further been argued that the capability to predict might be the foundation of what we understand as consciousness. Neuroscientist Anil Seth proposes that the brain is a prediction machine that is constantly generating our world or reality [13]. Seth suggests that when we agree with our generated world, we call it reality; when we do not, we call it delusion.

Seth’s insight is less concerned with whether our world is a (shared) construct and more with the proposition that prediction can determine our perception of the world and thus our apprehension of a shared reality: not only of the world around us, but of ourselves and others (our shared intersubjectivity). When we consider how surveillance systems operate to construct us as an object—both as a specific person(s) and as their correlate (the data double)—to generate further predictions (realities), the question becomes more nuanced and urgent.

In social media, what we experience is served to us based on what we seek, not at any one moment, but through our continuously “tracked” engagement. Social media users spend hours watching images, often without apparent purpose. Given these undirected and semiconscious media wanderings (doomscrolling), what kind of data double is constructed and what will subsequently be predicted and retrieved? Some of this behavior manifests as spam, which could be understood as an act of shared consumption, a process of exchange that objectifies the user. Hito Steyerl has observed that “image spam is one of the many dark matters of the digital world” [14].

Data collection occurs across written and spoken language, biometrics, geolocation, behavioral patterns, and vision, invoking Agre’s surveillance model. Image production and (re)circulation in social media relates closely to the accumulation of visual data (uploading, viewing, liking, remixing, and sharing), finding correspondences to Steyerl’s “poor image” [15]. Poor images, says Steyerl, “present a snapshot of the affective condition of the crowd, its neurosis, paranoia, and fear, as well as its craving for intensity, fun, and distraction” [16]. As the poor image represents a collective affective state, it assumes the representational status of the data double: the nexus of a network of correlates.

This is a hermeneutic dynamic, an endless algorithmic recirculation of messages, and a manifestation of the inter-subjectivity of our data doubles. However, these are deferred messages, eternally suspended abstractions not unlike the endless auditing by the narrator/messenger in Franz Kafka’s 1917 story The Great Wall of China [17], relaying descriptions of innumerable, endlessly forking paths (of construction of the Wall). The content of these messages is never delivered, only their envelopes. The delivery of the message is not the system’s raison d’être; rather, value is found in the surplus to be extracted from this constant recirculation. The platforms upon which these messages recirculate are constructed as a value chain predicated on the dynamic of an eternal deferral of meaning—a Kafkaesque information economy—ensuring a constant supply of excess value. Marshall McLuhan’s observation that the medium is the message continues to resonate in the age of social media [18].

Rita Raley has noted that “artists who appropriate dataveillance techniques and tools as a medium” help us critically imagine such apparatus as “Mirror Worlds” [19]. Given the value of the insights that might be gained through representing and reimagining a subject, in this case the data double, how might artists reflect upon this to reveal our second life as data doubles?

Keystone I, II, III, IV (2019) [20] by Rudolfo Quintas [21] (exhibited at the National Museum of Contemporary Art Chiado, Portugal [22]) interrogates the forms the data double might take (Fig. 1). In architecture, a keystone sustains the shape of an arch and locks its elements in place. In his exhibition essay, Mattia Tosti proposes data as the keystone of surveillance capitalism, and by implication foundational to the generation of data doubles and their recirculation as excess capital in the surveillance economy, echoing Zuboff. Presented as four sculptures, the installation Keystone I, II, III, IV collates networked communications over a three-month period, comprising “more than 100,000 tweets from the 100 most followed public figures in Portugal (politicians, newspapers, political parties, etc.)” [23].

Keystone is an example of how the two cultural models of privacy overlay and coexist. The reference to the keystone is a social and political metaphor that corresponds to the surveillance model, while the process of data capture employed in the work evokes Agre’s capture model. Each of the four sculptural elements in Keystone is connected to the others through a machine learning system trained to learn Portuguese. One, a suspended keystone, presents this as a word cloud of the most frequent (viewed, liked, and shared) Twitter (now “X”) feed topics, the font size of the words indicating their frequency. The remaining three keystones, placed on the floor, receive words from the suspended keystone and algorithmically generate different sentences composed of the same words. By recirculating the modified tweets, the resulting sentences contribute to an emergent collective voice, evolving in accordance with what are the most frequently mentioned topics of the day. The semantic content of the sentences might appear nonsensical by departing from typical sentence construction and lacking the usual context and nuance of words.

As a fundamental element maintaining a panoptic structure, the keystone may be considered an apt metaphor for our aggregated data doubles, traces of our collective selves resulting from individual online activities. Elements that constitute the sentences, the words that form the constellations, are recognizable but unreadable; poetic and nonsensical but sometimes carrying a sense of premonition, they do not accord with what we apprehend as typical human communication.

This uncanny quality of the sentences, composed by a machine learning agent from human action in a virtual world, contrasts with the familiarity evoked by the artwork Riccardo Uncut (2018) [24] by Eva and Franco Mattes [25] (hosted on the Whitney Museum’s ARTPORT platform [26]). This work reflects upon the visual representation of private lives and asks: How is our life’s memory constructed in an era of digital technologies and media networks?

Riccardo Uncut (Fig. 2) is composed of 3,000 images and videos from an individual’s mobile phone, forming 87 minutes of video. As the title suggests, the sequences follow the same order in which they were collected on the mobile phone, corresponding to a 13-year chronology of someone’s life (2004 to 2017). Composed of images and videos of varying resolutions and quality, Riccardo Uncut references historical forms of sharing personal and family images, such as photo albums, that were popular before digital technologies made the creation and sharing of recorded images relatively effortless, allowing for the almost uninterrupted recording of our contemporary lives. Private photographs and the memories they trigger emphasize an important purpose of photography: to construct a record of an event or moment, while recognizing the photograph’s status as a problematic mediated construct.

The contemporary sharing of photo albums, once historically restricted to family and friends within a private context, has become a public performativity for millions of people who capture and upload visual records of their lives every day. Riccardo Uncut, by contrast, offers access to recorded events from someone’s life with a familiarity reminiscent of past, more physical forms, of organizing photographic evidence. Riccardo, whoever they are, becomes someone the viewer comes to know intimately. However, this fabricated intimacy remains distant, as the memories that might be evoked by the photographs are not shared and thus never triggered. This constructed sense of intimacy is similarly characteristic of social networks.

Riccardo Uncut allows us to reflect on the evolution of photography by observing how personal imagery is constructed and shared through social networks. The use of mobile phones to register an(y) event has extended the scope of photography as a means to document and construct reality. Human agency is not yet completely erased from the process of taking a photograph—at least, not as significantly as Bolter and Grusin proposed in relation to digital graphics [27]—the mobile phone camera requires the agency of framing and pointing at the object or scenario to be captured, providing the immediacy of snapping a quick photograph anytime, anywhere, and rendering considerations of resolution, framing, and lighting largely automatic. These concerns have been transferred to the camera’s interface, where such adjustments have been incorporated into image previews. Capturing, editing, and sharing an image has become a simple, apparently immediate, task, accomplished with little more than the pressure of a thumb.

Photography has many of the properties of hypermediacy [28], foregrounding the process of mediation, while paradoxically providing a sense of immediacy concerning the performative construction of the recorded moment. This immediacy is part of the construct and thus not quite what it appears to be.

The algorithmic post-production of each photograph— whereby its luminosity and color can be adjusted, its frame cropped, and filters applied within moments of taking the shot—corresponds to Steyerl’s poor image. Riccardo Uncut is not a transmedia transfer from film to digital video but rather the construction of a digital video from historical modes of (re)presentation. Riccardo Uncut renders connections between the performative camera, the post-production of the photograph, and its distribution through social networks. As part of remediation, performing for the camera is learned by mimicking online exemplars and their strategies, enabled by instruments such as selfie sticks and photo-editing software. In a more recent project, Eva and Franco Mattes have acquired a different phone and presented the work Hannah Uncut (2021) [29]. In this work they present personal photographs that accentuate the process of image enhancement in their further mediation.

Through increasing circulation on social media platforms video has subsumed photography as a framework for contemporary normativity, structuring and informing how we see ourselves. This can be observed in a recent Dove soap television advertisement, Reverse Selfie (2021) [30]. The advertisement begins with an image of a mobile phone displaying a selfie of a young woman. Viewers follow, in reverse, the familiar process of posting (sharing), editing, and taking a photograph. Behind the phone is a woman who, as spectators, we follow through the process of (un)creating a persona for herself, mimicking conventional stereotypes of beauty to be rewarded with praise and likes, reinforcing the relationship between social media and normative ideas of beauty.

Through social media the sharing of personal images shifts from private to public, its reach extending to numerous algorithmically identified anonymous viewers. Without restrictive schedules and venues, screenings are no longer difficult to access. Like personal photographs, short films are pervasively and constantly available online, individual choice conditioned by collective preferences encapsulated through synoptic predictive algorithms (as employed on YouTube), fostering modes of viewing that supplant notions of public or private.

The Sprawl (Propaganda About Propaganda) (2016) [31] is an experimental documentary by Metahaven [32], co-commissioned and co-produced by Lighthouse (UK), that considers online spectatorship strategies (Fig. 3). It is made available on the web and structured in episodes hosted on YouTube. Each time the documentary’s website is accessed the order of episodes is algorithmically re-sorted. The episodes, of varied lengths, combine images from diverse sources (from the original documentary, archival imagery, 3D animations, and anime) that are sometimes edited like a videoclip, at others like an interview, and at yet others like a TV news broadcast.

Each episode, presented as a still image with a title and subtitle, alludes to specific memes, resembling a visually coherent collection of memes. Memes respond to a visual idea through a catchy image; like spam, replicated as Steyerl’s poor images, adapted and reformatted, they are fragmented. The viewer quickly identifies the ideas within the project as the components that comprise each episode, which together form a visual idea that requires little further engagement. This quick apprehension is apparent through a layer of graphic elements that joins all the parts, providing visual coherence. The attention previously required for apprehending a film has been replaced, as suggested by Steyerl, with distraction.

Memes require a priori cultural knowledge, without which the image-meme is nonsensical. Our shared cultural objects, such as a movie, a painting, or the news, are fundamental for memes to function, separating common sense—something we all agree on by definition—from non-sense: “A meme can tap into a collective memory and transform the ‘outcome’ of a commonly held starting point to different ends” [33].

The short film’s departure point is our shared knowledge of a philosophical and poetic order and world news. Gradual engagement with the documentary allows reflection on how we create fictions about ourselves and the world. Unlike the images from Riccardo Uncut, these images do not function as evidence of a lived reality. If we construct our profiles, so does everyone. If everyone is participating in the construction of fictions (memes) of themselves, why not accept those created by others? What if these fictions replace truth? What if these fictions assume public, international proportions? Memes can be jokes, signifiers, or opinions. In the work of Metahaven, they are political tools.

The unconscious scrolling, selection, and resulting continuous circulation of memes contribute to the formation of correlates; that is, our and other’s data double. As a result, due to our role in the co-construction of our digital environment, we (vaguely) recognize the heterogeneous cultural elements comprising the meme. This is not recognition of what the cultural objects signify of a person. Rather, like the correlate, the meme represents a connection between ourselves and our data double. The collective memory that permits the recognition of a given meme results from the intertwined connection to Seth’s concept of how we shape our reality.

There are numerous artists working with the themes of big data and surveillance. We can take note of precursor artists, such as Lynn Hershman Leeson and Antonio Muntadas; contemporary artists Memo Akten, Evan Roth, and James Bridle; or consider the work “Nakamoto (The Proof)” (2014) by Émile Brout and Maxime Marion, which constructs an identity for the (possibly fictional) inventor of Bitcoin by making incursions into the dark web to correlate traces of his almost nonexistent data double [34].

Notable for the scope and ambition of its engagement with surveillance systems and big data, the installation EXIT (2008–2019), commissioned by the Fondation Cartier, was inspired by an idea of Paul Virilio and developed by architects Diller Scofidio + Renfro, Mark Hansen, Laura Kurgan, and Ben Rubin in collaboration with Robert Gerard Pietrusko and Stewart Smith (Fig. 4). The work is an immersive multimedia environment composed of several video projections displaying datasets that map contemporary global events, including the movement of refugees, their financial remittances across borders, the effects of climate change, and real-time stock exchange data. The work creates an impression of our planet traced through a multitude of data modalities. Its effect is overwhelming in part because of its disturbing “corporate cool” or technocratic aesthetic. If EXIT is considered an envisaging of the World’s own data double, then EXIT becomes a data double presented in as much a state of crisis as its correlate:

With the world presented, in EXIT, as its data double, we gain a bird’s-eye view of the subject constituted as Richard Brautigan’s “cybernetic ecology,” but one where we are not “free of our labors.”

Brautigan’s vision of a future free of labor is evoked in Nadim Choufi’s film The Sky Oscillates Between Eternity and Its Immediate Consequences (2020). This film is a portal to a future where machines control a perfectly organized colony in the desert. In this algorithmically constructed environment, diversity and diference are considered a threat to collective survival. Providing one’s personal data as a means to contribute to the economy is mandatory. Despite this work of machines, Choufi asks: “Isn’t it chance that makes our essence?” [36].

We live in an era where we routinely accept, without review, lengthy, incomprehensible legal permissions to access media platforms, information, and software. This is an information economy predicated on seduction, where big data amplifies our desire for a perfect world; a world where people, individually and collectively, are rendered as excess capital—not producers of value, but value itself (a raw product, our data). The processes by which this raw material (humanity) is transformed into currency are in plain sight, captured and rendered as data doubles: We are no longer subjects, but objects, all watched over by our Data Double.