Assuming that patterns in memory are represented as two-dimensional arrays of local features, just as they are in primary visual cortices, pattern recognition can take the form of elastic graph matching (Lades et al., 1993). Neural implementation of this may be based on preorganized fiber projections that can be activated rapidly with the help of control units (Wolfrum, Wolff, Lücke, & von der Malsburg, 2008). Each control unit governs a set of projection fibers that form part of a coherent mapping. We describe a mathematical model for the ontogenesis of the underlying connectivity based on a principle of network self-organization as described by the Häussler system (Häussler & von der Malsburg, 1983), modified to be sensitive to pattern similarity and to support formation of multiple mappings, each under the command of a control unit. The process takes the form of a soft-winner-take-all, where units compete for the representation of maps. We show simulations for invariant point-to-point and feature-to-feature mappings.