Yuanmingyuan was called “Garden of All Gardens” before it was looted and burnt down by the Anglo-French allied forces in 1860. Nowadays there are only stone ruins that used to be parts of ancient buildings. The Digital Yuanmingyuan project is designed to provide visitors with the visualization of virtual reconstructions superimposed upon the natural field of the ruins using a personal AR (Augmented Reality) system. It can not only preserve the current appearance of the ruins, but also exhibit the original exquisite architecture of Yuanmingyuan. The technical difficulties of the system are analyzed, and the design of the system hardware and the tracking algorithm are discussed. A prototype of the proposed system is developed and the initial result of restoring one typical scene in Yuanmingyuan is presented.