This paper describes the author's own personal experiences, experiments, and lifelong narrative of inventing, designing, building, and living with a variety of body-borne computer-based visual information capture and mediation devices. The emphasis is not just on the devices themselves, but on certain social, privacy, ethical, and legal questions and challenges that have arisen from actual experiences with lifelong video capture, processing, transmission, and dissemination in a variety of different everyday cultural settings over the past 30 years. The most interesting of these accidentally-found questions pertain to: (1) inverse surveillance (body-borne audiovisual and other sensor capture, storage, recall, and processing, known in the research literature as “sousveillance”); and (2) the epistemology of freewill and metaphysics of choice that seems to arise from an apparent reversal of the now pervasive and ubiquitous notion of surveillance. Extrapolating from these lessons, several hypotheses are presented, including: (1) sousveillance, like surveillance, will be driven by rapid development of new technology, leaving legal frameworks lagging behind technology; (2) the growth of sousveillance will accelerate greatly when implementations come with other non-sousveillance uses (e.g., camera phones because of their strategic ambiguity with regard to whether they are being used to take a picture or for just a voice call); (3) legal frameworks will tend to support rather than oppose sousveillance; (4) such legal protections will favor video sousveillance over video surveillance just as they now favor audio sousveillance over audio surveillance; (5) such legal protections will emerge first for the disabled (e.g., the visually impaired); and will then expand to encompass other legitimate and beneficial uses of sousveillance (personal safety, evidence gathering, etc.); (6) a person wishing to do lifelong sousveillance is deserving of certain legal protections liabilizing others who might attempt to disrupt continuity of evidence.

Diminished reality is as important as augmented reality, and both are possible with a device called the Reality Mediator . Over the past two decades, we have designed, built, worn, and tested many different embodiments of this device in the context of wearable computing. Incorporated into the Reality Mediator is an “EyeTap” system, which is a device that quantifies and resynthesizes light that would otherwise pass through one or both lenses of the eye(s) of a wearer. The functional principles of EyeTap devices are discussed, in detail. The EyeTap diverts into a spatial measurement system at least a portion of light that would otherwise pass through the center of projection of at least one lens of an eye of a wearer. The Reality Mediator has at least one mode of operation in which it reconstructs these rays of light, under the control of a wearable computer system. The computer system then uses new results in algebraic projective geometry and comparametric equations to perform head tracking, as well as to track motion of rigid planar patches present in the scene. We describe how our tracking algorithm allows an EyeTap to alter the light from a particular portion of the scene to give rise to a computer-controlled, selectively mediated reality. An important difference between mediated reality and augmented reality includes the ability to not just augment but also deliberately diminish or otherwise alter the visual perception of reality. For example, diminished reality allows additional information to be inserted without causing the user to experience information overload. Our tracking algorithm also takes into account the effects of automatic gain control, by performing motion estimation in both spatial as well as tonal motion coordinates.

Wearable computing moves computation from the desktop to the user. We are forming a community of networked, wearable-computer users to explore, over a long period, the augmented realities that these systems can provide. By adapting its behavior to the user's changing environment, a body-worn computer can assist the user more intelligently, consistently, and continuously than a desktop system. A text-based augmented reality, the Remembrance Agent, is presented to illustrate this approach. Video cameras are used both to warp the visual input (mediated reality) and to sense the user's world for graphical overlay. With a camera, the computer could track the user's finger to act as the system's mouse; perform face recognition; and detect passive objects to overlay 2.5D and 3D graphics onto the real world. Additional apparatus such as audio systems, infrared beacons for sensing location, and biosensors for learning about the wearer's affect are described. With the use of input from these interface devices and sensors, a long-term goal of this project is to model the user's actions, anticipate his or her needs, and perform a seamless interaction between the virtual and physical environments.