Social photo-sharing sites like Flickr contain vast amounts of latent information about the world and human behavior. The authors describe their recent work in building automatic algorithms that analyze large collections of imagery in order to extract some of this information. At a global scale, geo-tagged photographs can be used to identify the most photographed places on Earth, as well as to infer the names and visual representations of these places. At a local scale, the authors build detailed 3D models of a scene by combining information from thousands of 2D photographs taken by different people and from different vantage points.