In evolutionary robotics, controllers are often represented as networks. Modularity is a desirable trait of such networks because modular networks are resistant to catastrophic forgetting and tend to have less connections than nonmodular ones. However, these advantages can only be realized if the control task is solvable by a modular network, and for any given practical task the control task depends on the choice of the robot’s morphology. Here we provide an example of a task solvable by robots with two different morphologies. We consider the most extreme kind of modularity – disconnectedness – and show that with the first morphology the task can be solved by a disconnected controller with few connections. On the other hand, the second morphology makes the task provably impossible for disconnected controllers and requires about three times more connections. For this morphology, most controllers that partially solve the task constitute local optima, forming an extremely deceptive fitness landscape. We show empirically that in this case a connection cost-based evolutionary algorithm for evolving modular controllers is greatly slowed down compared to the first morphology’s case. Finally, this performance gap increases as the task is scaled up. These results show that the morphology may be a major factor determining the performance of controller optimization. Although in our task the optimal morphology is obvious to a human designer, we hypothesize that as evolutionary robotics is scaled to more sophisticated tasks the optimization of morphology alongside the control might become a requirement for evolving modular controllers.