The addition of parasites to a host population can drive an escalation in the host population's phenotypic complexity – even in the absence of a direct fitness advantage for this increase. Parasites restrict certain regions of the genotype space, decreasing the fitness and the probability of survival of particular host phenotypes. While many artificial life frameworks model a direct correlation between genotype and fitness, the structure of genotype-phenotype maps can have important effects on evolutionary dynamics. Using a simple coarse-grained model for phenotypic transitions during evolution, we show that the escalation in phenotypic complexity under neutral co-evolution is dependent on the structure of the genotype-phenotype map. We discuss these results using the metaphor of evolutionary spandrels and highlight how these structural considerations might allow us to capture biological phenomena more accurately.