Effect of Depth and Width on Local Minima in Deep Learning

Deep learning with neural networks has been a significant practical success in many fields, including computer vision, machine learning, and artificial intelligence. Along with its practical success, deep learning has been theoretically analyzed and shown to be attractive in terms of its expressive power. For example, neural networks with one hidden layer can approximate any continuous function (Leshno, Lin, Pinkus, & Schocken, 1993; Barron, 1993), and deeper neural networks enable us to approximate functions of certain classes with fewer parameters (Montufar, Pascanu, Cho, & Bengio, 2014; Livni, Shalev-Shwartz, & Shamir, 2014; Telgarsky, 2016). However, training deep learning models requires us to work with a seemingly intractable problem: nonconvex and high-dimensional optimization. Finding a global minimum of a general nonconvex function is NP-hard (Murty & Kabadi, 1987), and nonconvex optimization to train certain types of neural networks is also known to be NP-hard (Blum & Rivest, 1992). These hardness results pose a serious concern only for high-dimensional problems, because global optimization methods can efficiently approximate global minima without convexity in relatively low-dimensional problems (Kawaguchi, Kaelbling, & Lozano-Pérez, 2015).

A hope is that beyond the worst-case scenarios, practical deep learning allows some additional structure or assumption to make nonconvex high-dimensional optimization tractable. Recently, it has been shown with strong simplification assumptions that there are novel loss landscape structures in deep learning optimization that may play a role in making the optimization tractable (Dauphin et al., 2014; Choromanska, Henaff, Mathieu, Ben Arous, & LeCun, 2015; Kawaguchi, 2016). Another key observation is that if a neural network is strongly overparameterized so that it can memorize any data set of a fixed size, then all stationary points (including all local minima and saddle points) become global minima, with some nondegeneracy assumptions. This observation was explained by Livni et al. (2014) and further refined by Nguyen and Hein (2017, 2018). However, these previous results (Livni et al., 2014; Nguyen and Hein, 2017, 2018) require strong overparameterization by assuming not only that a network's width is larger than the data set size but also that optimizing only a single layer (the last layer or some hidden layer) can memorize any data set based on an assumed condition on the rank or nondegeneracy of other layers.

In this letter, we analyze the effects of depth and width on the values of local minima, without the strong overparameterization and simplification assumptions in the literature. As a result, we prove quantitative upper bounds on the quality of local minima, which shows that the values of local minima of neural networks are guaranteed to be no worse than the globally optimal values of corresponding classical machine learning models, and the guarantee can improve as depth and width increase.

This section defines the optimization problem considered in this letter and introduces the basic notation.

Let $x∈Rdx$ and $y∈Rdy$ be an input vector and a target vector, respectively. Let ${(xi,yi)}i=1m$ be a training data set of size $m$⁠. Given a set of $n$ matrices or vectors ${M(j)}j=1n$⁠, define $[M(j)]j=1n:=M(1)M(2)⋯M(n)$ to be a block matrix of each column block being $M(1),M(2),⋯,M(n)$⁠. Define the training data matrices as $X:=([xi]i=1m)⊤∈Rm×dx$ and $Y:=([yi]i=1m)⊤∈Rm×dy$⁠.

The output prediction matrix $Y^(X,θ)∈Rm×dy$ is specified for shallow networks with rectified linear units (ReLUs) in section 3 and generalized to deep nonlinear neural networks in section 4.

Define $P[M]$ to be the orthogonal projection matrix onto the column space (or range space) of a matrix $M$⁠. Let $PN[M]$ be the orthogonal projection matrix onto the null space (or kernel space) of a matrix $M⊤$⁠. For a matrix $M∈Rd×d'$⁠, we denote the standard vectorization of the matrix $M$ as $vec(M)=[M1,1,…,Md,1,M1,2,⋯,Md,2,…,M1,d',…,Md,d']T$⁠.

Before presenting our main results for deep nonlinear neural networks, this section provides the results for shallow networks with a single hidden layer (or three-layer networks with the input and output layers) and scalar-valued output (i.e., $dy=1$⁠) to illustrate some of the ideas behind the discussed effects of the depth and width on local minima.

Under this setting, proposition 1 provides an equation that holds at local minima and illustrates the effect of width for shallow ReLU neural networks.

Proposition 1 is an immediate consequence of our general result (see theorem 1) in the next section (the proof is provided in section A.1). In the rest of this section, we provide a proof sketch of proposition 1.

From equation 2.2 in proposition 1, the loss $L(θ)$ at differentiable local minima is expected to tend to get smaller as the width of the hidden layer $d$ gets larger. To further support this theoretical observation, this section obtains a probabilistic upper bound on the loss $L(θ)$ for white noise data by fixing the activation patterns $Λ1,k$ for $k∈{1,2,⋯,d}$ and assuming that the data matrix $[XY]$ is a random gaussian matrix, with each entry having mean zero and variance one.

This definition of $Λii1,k$ generalizes the corresponding definition in section 3.1. Proposition 1 holds for this generalized activation pattern by simply replacing the previous definition of $Λii1,k$ by this more general definition. This can be seen from the proof sketch in section 3.1 and is later formalized in the proof of theorem 1.

We denote the vector consisting of the diagonal entries of $Λ1,k$ by $Λk∈Rm$ for $k∈{1,2,⋯,d}$⁠. Define the activation pattern matrix as $Λ:=[Λk]k=1d∈Rm×d$⁠. For any index set $I⊆{1,2,⋯,m}$⁠, let $ΛI$ denote the submatrix of $Λ$ that consists of its rows of indices in $I$⁠. Let $smin(ΛI)$ be the smallest singular value of $ΛI$⁠.

Proposition 2 proves that $L(θ)≈(1-dxd/m)∥Y∥22/2$ in the regime $dxd≪m$⁠, and $L(θ)=0$ in the regime $dxd≫m$⁠, under the corresponding conditions on $Λ$⁠; that is, $smin(ΛI)≥δ$ for any index set $I⊆{1,2,⋯,m}$ such that $|I|≥m/2$ in the regime $dxd≪m$⁠, and $|I|≤d/2$ in the regime $dxd≫m$⁠. This supports our theoretical observation that increasing width helps improve the quality of local minima.

The proof of proposition 2 is provided in appendix B. In that proof, we first rewrite the loss $L(θ)$ as the projection of $Y$ onto the null space of an $m×dd0$ matrix $D˜$⁠, with an explicit expression in terms of the activation pattern matrix $Λ$ and the data matrix $X$⁠. By our assumption, the data matrix $X$ is a random gaussian matrix. The projection matrix $D˜$ is also a random matrix. Proposition 2 then boils down to understanding the rank of the projection matrix $D˜$⁠, and we proceed to show that $D˜$ has the largest possible rank, $min{dd0,m}$⁠, with high probability. In fact, we derive quantitative estimates on the smallest singular value of $D˜$⁠. The main difficulties are that the columns of the matrix $D˜$ are correlated and variances of different entries vary. Our approach to obtain quantitative estimates on the smallest singular value of $D˜$ combines the epsilon net argument with an iterative argument.

In the regime $dd0≫m$⁠, results similar to proposition 2ii were obtained under certain diversity assumptions on the entries of the weight matrices in a previous study (Xie, Liang, & Song, 2017). When compared with the previous study (Xie et al., 2017), proposition 2 specifies precise relations between the size $dd0$ of the neural network and the size $m$ of the data set and also holds true in the regime $dd0≪m$⁠. Moreover, our proof arguments for proposition 2ii are different. Xie et al. (2017), under the assumption that $dd0≫m$⁠, show that $D˜D˜T$ is close to its expectation in the sense of spectral norm. As a consequence, the lower bound of the smallest eigenvalue of $E[D˜D˜T]$ gives the lower bound for the smallest singular value of $D˜$⁠.

However, proposition 2 assumes a gaussian data matrix, which may be a substantial limitation. The proof of proposition 2 relies on the concentration properties of gaussian distribution. Whereas a similar proof would be able to extend proposition 2 to a nongaussian distribution with these properties (e.g., distributions with subgaussian tails), it would be challenging to use a similar proof for a general distribution without the properties similar to those.

Let $H$ be the number of hidden layers and $dl$ be the width (or, equivalently, the number of units) of the $l$th hidden layer. To theoretically analyze concrete phenomena, the rest of this letter focuses on fully connected feedforward networks with various depths $H≥1$ and widths $dl≥1$⁠, using rectified linear units (ReLUs), leaky ReLUs, and absolute value activations, evaluated with the squared loss function. In the rest of this letter, the (finite) depth $H$ can be arbitrarily large and the (finite) widths $dl$ can arbitrarily differ among different layers.

This definition of $Λiil,k$ generalizes the corresponding definition in section 3. Let $Id$ be the identity matrix of size $d$ by $d$⁠. Define $M⊗M'$ to be the Kronecker product of matrices $M$ and $M'$⁠. Given a matrix $M$⁠, $M·,j$ and $Mi,·$ denote the $j$th column vector of $M$ and the $i$th row vector of $M$⁠, respectively.

For the standard deep nonlinear neural networks, theorem 1 provides an equation that holds at local minima and illustrates the effect of depth and width. Let $dl':=dl$ for all $l∈{1,⋯,H}$ and $dH+1':=1$⁠.

The complete proof of theorem 1 is provided in section A.1. Theorem 1 is a generalization of proposition 1. Accordingly, its proof follows the proof sketch presented in the previous section for proposition 1.

Unlike previous studies (Livni et al., 2014; Nguyen & Hein, 2017, 2018), theorem 1 requires no overparameterization such as $dl≥m$⁠. Instead, it provides quantitative gradual effects of depth and width on local minima, from no overparameterization to overparameterization. Notably, theorem 1 shows the effect of overparameterization in terms of depth as well as width, which also differs from the results of previous studies that consider overparameterization in terms of width (Livni et al., 2014; Nguyen & Hein, 2017, 2018).

The proof idea behind these previous studies with strong overparameterization is captured in the discussion after equation 3.3—with strong overparameterization such that $dl≥m$ and $rank(D(1))≥m$⁠, $∇vec(W)Y^(X,θ)∈Rdl×m$ is left-invertible and hence every local minimum is a global minimum with zero training error. Here, $rank(M)$ represents the rank of a matrix $M$⁠. The proof idea behind theorem 1 differs from those as shown in section 3.1. What is still missing in theorem 1 is the ability to provide a prior guarantee on $L(θ)$ without strong overparameterization, which is addressed in sections 3.2 and 5 for some special cases but left as an open problem for other cases.

In the synthetic data set, the data points ${(xi,yi)}i=1m$ were randomly generated by a ground-truth, fully connected feedforward neural network with $H=7$⁠, $dl=50$ for all $l∈{1,2,⋯,H}$⁠, tanh activation function, $(x,y)∈R10×R$ and $m=5000$⁠. MNIST (LeCun, Bottou, Bengio, & Haffner, 1998), a popular data set for recognizing handwritten digits, contains 28 $×$ 28 gray-scale images. The CIFAR-10 (Krizhevsky & Hinton, 2009) data set consists of 32 $×$ 32 color images that contain different types of objects such as “airplane,” “automobile,” and “cat.” The Street View House Numbers (SVHN) data set (Netzer et al., 2011) contains house digits collected by Google Street View, and we used the 32 $×$ 32 color image version for the standard task of predicting the digits in the middle of these images. In order to reduce the computational cost, for the image data sets (MNIST, CIFAR-10, and SVHN), we center-cropped the images (⁠$24×24$ for MNIST and $28×28$ for CIFAR-10 and SVHN), then resized them to smaller gray-scale images (⁠$8×8$ for MNIST and $12×12$ for CIFAR-10 and SVHN), and used randomly selected subsets of the data sets with size $m=10,000$ as the training data sets.

For all the data sets, the network architecture was fixed to be a fully connected feedforward network with the ReLU activation function. For each data set, the values of $J(θ)$ were computed with initial random weights drawn from a normal distribution with zero mean and normalized standard deviation (⁠$1/dl$⁠) and with trained weights at the end of 40 training epochs. (Additional experimental details are presented in appendix C.)

Figure 1 shows the results with the synthetic data set, as well as the MNIST, CIFAR-10, and SVHN data sets. As it can be seen, the values of $J(θ)$ tend to decrease toward zero (and hence the global minimum value), as the width or depth of neural networks increases. In theory, the values of $J(θ)$ may not improve as much as desired along depth and width if representations corresponding to each unit and each layer are redundant in the sense of linear dependence of the columns of $Dkl(l)(θ)$ (see theorem 1). Intuitively, at initial random weights, one can mitigate this redundancy due to the randomness of the weights, and hence a major concern is whether such redundancy arises and $J(θ)$ degrades along with training. From Figure 1, it can be also noticed that the values of $J(θ)$ tend to decrease along with training. These empirical results partially support our theoretical observation that increasing the depth and width can improve the quality of local minima.

Given the scarcity of theoretical understanding of the optimality of deep neural networks, Goodfellow, Bengio, and Courville (2016) noted that it is valuable to theoretically study simplified models: deep linear neural networks. For example, Saxe, McClelland, and Ganguli (2014) empirically showed that in terms of optimization, deep linear networks exhibited several properties similar to those of deep nonlinear networks. Following these observations, the theoretical study of deep linear neural networks has become an active area of research (Kawaguchi, 2016; Hardt & Ma, 2017; Arora, Cohen, Golowich, & Hu, 2018; Arora, Cohen, & Hazan, 2018), as a step toward the goal of establishing the optimization theory of deep learning.

As another step toward the goal, this section discards the strong linearity assumption and considers a locally induced nonlinear-linear structure in deep nonlinear networks with the piecewise linear activation functions such as ReLUs, leaky ReLUs, and absolute value activations.

In this section, we describe how a standard deep nonlinear neural network can induce nonlinear-linear structure. The nonlinear-linear structure considered in this letter is defined in definition 1: condition i simply defines the index subsets $S(l)$ that pick out the relevant subset of units at each layer $l$⁠, condition ii requires the existence of $n$ linearly acting units, and condition iii imposes weak separability of edges.

Given a training input data set $X$⁠, let $Θn,t$ be the set of all parameter vectors that induce $(n,t)$ weakly separated linear units on the training input data set $X$ that defines the total loss $L(θ)$ in equation 2.1. For standard deep nonlinear neural networks, all parameter vectors $θ$ are in $ΘdH+1,H$⁠, and some parameter vectors $θ$ are in $Θn,t$ for different values of $(n,t)$⁠. Figure 2 a illustrates locally induced structures for $θ∈Θ1,0$⁠. For a parameter $θ$ to be in $Θn,t$⁠, definition 1 requires only the existence of a portion $n/dl$ of units to act linearly on the particular training data set merely at the particular $θ$⁠. Thus, all units can be nonlinear, act nonlinearly on the training data set outside of some parameters $θ$⁠, and operate nonlinearly always on other inputs $x$—for example, in a test data set or a different training data set. The weak separability requires that the edges going from the $n$ units to the rest of the network are negligible. The weak separability does not require the $n$ units to be separated from the rest of the neural network.

Here, a neural network with $θ∈Θn,t$ can be a standard deep nonlinear neural network (without any linear units in its architecture), a deep linear neural network (with all activation functions being linear), or a combination of these cases. Whereas a standard deep nonlinear neural network can naturally have parameters $θ∈Θn,t$⁠, it is possible to guarantee all parameters $θ$ to be in $Θn,t$ with desired $(n,t)$ simply by using corresponding network architectures. For standard deep nonlinear neural networks, one can also restrict all relevant convergent solution parameters $θ$ to be in $Θn,t$ by using some corresponding learning algorithms. Our theoretical results hold for all of these cases.

We state our main theoretical result in theorem 2 and corollary 1; a simplified statement is presented in remark 1. Here, a classical machine learning method, basis function regression, is used as a baseline to be compared with neural networks. The global minimum value of basis function regression with an arbitrary basis matrix $M(X)$ is $infR12∥M(X)R-Y∥F2$⁠, where the basis matrix $M(X)$ does not depend on $R$ and can represent nonlinear maps, for example, by setting $M=([φ(xi)]i=1m)⊤∈Rm×dφ$ with any nonlinear basis functions $φ$ and any finite $dφ$⁠. In theorem 2, the expression $PNΦ(S)Y$ represents the projection of $Y$ onto the null space of $(Φ(S))⊤$⁠, which is also (⁠$Y$—the projection of $Y$ onto the column space of $Φ(S)$⁠). Given matrices $(M(j))j∈S$ with a sequence $S=(s1,s2,⋯,sn)$⁠, define $[M(j)]j∈S:=M(s1)M(s2)⋯M(sn)$ to be a block matrix with columns being $M(s1),M(s2),⋯,M(sn)$⁠. Let $S⊆(s1,s2,⋯,sn)$ denote a subsequence of $(s1,s2,⋯,sn)$⁠.

The proof of theorem 2 is provided in section A.2. The proof is based on the combination of the idea presented in section 3.1 and perturbations of a local minimum candidate. That is, if a $θ$ is a local minimum, then the $θ$ is a global minimum within a local region (i.e., a neighborhood of $θ$⁠). Thus, after perturbing $θ$ as $θ'=θ+Δθ$ such that $∥Δθ∥$ is sufficiently small (so that $θ'$ stays in the local region) and $L(θ')=L(θ)$⁠, the $θ'$ must be still a global minimum within the local region and, hence, the $θ'$ is also a local minimum. The proof idea of theorem 2 is to apply the proof sketch in section 3.1 to not only a local minimum candidate $θ$ but also its perturbations $θ'=θ+Δθ$⁠.

In terms of overparameterization, theorem 2 states that local minima of deep neural networks are as good as global minima of the corresponding basis function regression even without overparameterization, and overparameterization helps to further improve the guarantee on local minima. The effect of overparameterization is captured in both the first and second terms on the right-hand side of equation 5.1. As depth and width increase, the second term tends to increase, and hence the guarantee on local minima can improve. Moreover, as depth and width increase (for some of $t+1,t+2,⋯,H$th layers in theorem 2), the first term tends to decrease and the guarantee on local minima can also improve. For example, if $[Φ(l)]l=tH$ has rank at least $m$⁠, then the first term is zero and, hence, every local minimum is a global minimum with zero loss value. As a special case of this example, since every $θ$ is automatically in $ΘdH+1,H$⁠, if $Φ(H)$ is forced to have rank at least $m$⁠, every local minimum becomes a global minimum for standard deep nonlinear neural networks, which coincides with the observation about overparameterization by Livni et al. (2014).

Without overparameterization, theorem 2 also recovers one of the main results in the literature of deep linear neural networks as a special case—that is, every local minimum is a global minimum. If $dH+1≤min{dl:1≤l≤H}$⁠, every local minimum $θ$ for deep linear networks is differentiable and in $ΘdH+1,0$⁠, and hence theorem 1 yields that $L(θ)≤12∥PN[X]Y∥F2$⁠. Because $12∥PN[X]Y∥F2$ is the global minimum value, this implies that every local minimum is a global minimum for deep linear neural networks.

Corollary 1 states that the same conclusion and discussions as in theorem 2 hold true even if we fix the edges in condition iii in definition 1 to be zero (by removing them as an architectural design or by forcing it with a learning algorithm) and consider optimization problems only with remaining edges.