Some of the great physicists' belief in the existence of a connection between the aesthetical features of a theory (such as beauty and simplicity) and its truth is still one of the most intriguing issues in the aesthetics of science. In this paper I explore the philosophical credibility of a version of this thesis, focusing on the connection between the mathematical beauty and simplicity of a theory and its truth. I discuss a heuristic interpretation of this thesis, attempting to clarify where the appeal of this Pythagorean view comes from and what are the arguments favoring its acceptance or rejection. Along the way, I sketch the historical context in which this heuristic interpretation gained credibility (the quantum crisis in physics in the 1920s), as well as the more general implications of this thesis for physicists' metaphysical outlook.