The authors propose a deep neural network–based algorithm to automatically generate portrait map art (PMA), a modern art form created by British portrait artist Ed Fairburn. The authors formulate the generation of PMA as an adaptive dual-to-single image translation problem. The authors’ proposed model analyzes the appearance of one portrait and one map image using two encoder networks and utilizes their hidden encodings as representations of the portrait and map image to generate new PMA using a decoder network. An adaptive style harmonization module is proposed to fuse the two hidden encodings. Optimized by cycle-consistency constraint, the model can produce new PMA images without baselines.