States in hypercube interior get pulled into spurious basins of attraction. PGD is in green and Multiplicative Weights in orange. Network is initialized at a distance $θ$ from the center of the simplex (see equation 6.14), and allowed to converge. The $y$-axis is the accuracy of the factorization implied by the converged state. Triangles indicate initialization slightly away from $af★$ toward any of the other simplex vertices, which is most directions in the space. These initial states get quickly pulled into a spurious basin of attraction.