Every Local Minimum Value Is the Global Minimum Value of Induced Model in Nonconvex Machine Learning

Kawaguchi, Kenji; Huang, Jiaoyang; Kaelbling, Leslie Pack

doi:10.1162/neco_a_01234

Abstract

For nonconvex optimization in machine learning, this article proves that every local minimum achieves the globally optimal value of the perturbable gradient basis model at any differentiable point. As a result, nonconvex machine learning is theoretically as supported as convex machine learning with a handcrafted basis in terms of the loss at differentiable local minima, except in the case when a preference is given to the handcrafted basis over the perturbable gradient basis. The proofs of these results are derived under mild assumptions. Accordingly, the proven results are directly applicable to many machine learning models, including practical deep neural networks, without any modification of practical methods. Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights. A special case of our results also contributes to the theoretical foundation of representation learning.

1 Introduction

Deep learning has achieved considerable empirical success in machine learning applications. However, insufficient work has been done on theoretically understanding deep learning, partly because of the nonconvexity and high-dimensionality of the objective functions used to train deep models. In general, theoretical understanding of nonconvex, high-dimensional optimization is challenging. Indeed, finding a global minimum of a general nonconvex function (Murty & Kabadi, 1987) and training certain types of neural networks (Blum & Rivest, 1992) are both NP-hard. Considering the NP-hardness for a general set of relevant problems, it is necessary to use additional assumptions to guarantee efficient global optimality in deep learning. Accordingly, recent theoretical studies have proven global optimality in deep learning by using additional strong assumptions such as linear activation, random activation, semirandom activation, gaussian inputs, single hidden-layer network, and significant overparameterization (Choromanska, Henaff, Mathieu, Ben Arous, & LeCun, 2015; Kawaguchi, 2016; Hardt & Ma, 2017; Nguyen & Hein, 2017, 2018; Brutzkus & Globerson, 2017; Soltanolkotabi, 2017; Ge, Lee, & Ma, 2017; Goel & Klivans, 2017; Zhong, Song, Jain, Bartlett, & Dhillon, 2017; Li & Yuan, 2017; Kawaguchi, Xie, & Song, 2018; Du & Lee, 2018).

A study proving efficient global optimality in deep learning is thus closely related to the search for additional assumptions that might not hold in many practical applications. Toward widely applicable practical theory, we can also ask a different type of question: If standard global optimality requires additional assumptions, then what type of global optimality does not? In other words, instead of searching for additional assumptions to guarantee standard global optimality, we can also search for another type of global optimality under mild assumptions. Furthermore, instead of an arbitrary type of global optimality, it is preferable to develop a general theory of global optimality that not only works under mild assumptions but also produces the previous results with the previous additional assumptions, while predicting new results with future additional assumptions. This type of general theory may help not only to explain when and why an existing machine learning method works but also to predict the types of future methods that will or will not work.

As a step toward this goal, this article proves a series of theoretical results. The major contributions are summarized as follows:

For nonconvex optimization in machine learning with mild assumptions, we prove that every differentiable local minimum achieves global optimality of the perturbable gradient basis model class. This result is directly applicable to many existing machine learning models, including practical deep learning models, and to new models to be proposed in the future, nonconvex and convex.
The proposed general theory with a simple and unified proof technique is shown to be able to prove several concrete guarantees that improve or complement several state-of-the-art results.
In general, the proposed theory allows us to see the effects of the design of models, methods, and assumptions on the optimization landscape through the lens of the global optima of the perturbable gradient basis model class.

Because a local minimum $θ$ in $R^{d_{θ}}$ only requires the $θ$ to be locally optimal in $R^{d_{θ}}$ ⁠, it is nontrivial that the local minimum is guaranteed to achieve the globally optimality in $R^{d_{θ}}$ of the induced perturbable gradient basis model class. The reason we can possibly prove something more than many worst-case results in general nonconvex optimization is that we explicitly take advantage of mild assumptions that commonly hold in machine learning and deep learning. In particular, we assume that an objective function to be optimized is structured with a sum of weighted errors, where each error is an output of composition of a loss function and a function of a hypothesis class. Moreover, we make mild assumptions on the loss function and a hypothesis class, all of which typically hold in practice.

2 Preliminaries

This section defines the problem setting and common notation.

2.1 Problem Description

Let

x \in X

and

y \in Y

be an input vector and a target vector, respectively. Define

{((x_{i}, y_{i}))}_{i = 1}^{m}

as a training data set of size

m

⁠. Let

θ \in R^{d_{θ}}

be a parameter vector to be optimized. Let

f (x; θ) \in R^{d_{y}}

be the output of a model or a hypothesis, and let

ℓ : R^{d_{y}} \times Y \to R_{\geq 0}

be a loss function. Here,

d_{θ}, d_{y} \in N_{> 0}

⁠. We consider the following standard objective function

L

to train a model

f (x; θ)

⁠:

L (θ) = \sum_{i = 1}^{m} λ_{i} ℓ (f (x_{i}; θ), y_{i}) .

This article allows the weights

λ_{1}, \dots, λ_{m} > 0

to be arbitrarily fixed. With

λ_{1} = \dots = λ_{m} = \frac{1}{m}

⁠, all of our results hold true for the standard average loss

L

as a special case.

2.2 Notation

Because the focus of this article is the optimization of the vector

θ

⁠, the following notation is convenient:

ℓ_{y} (q) = ℓ (q, y)

and

f_{x} (q) = f (x; q)

⁠. Then we can write

L (θ) = \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) = \sum_{i = 1}^{m} λ_{i} (ℓ_{y_{i}} \circ f_{x_{i}}) (θ) .

We use the following standard notation for differentiation. Given a scalar-valued or vector-valued function $ϕ : R^{d} \to R^{d^{'}}$ with components $ϕ = (ϕ_{1}, \dots, ϕ_{d^{'}})$ and variables $(v_{1}, \dots, v_{\bar{d}})$ ⁠, let $\partial_{v} ϕ : R^{d} \to R^{d^{'} \times \bar{d}}$ be the matrix-valued function with each entry ${(\partial_{v} ϕ)}_{i, j} = \frac{\partial ϕ_{i}}{\partial v_{j}}$ ⁠. Note that if $ϕ$ is a scalar-valued function, $\partial_{v} ϕ$ outputs a row vector. In addition, $\partial ϕ = \partial_{v} ϕ$ if $(v_{1}, \dots, v_{d})$ are the input variables of $ϕ$ ⁠. Given a function $ϕ : R^{d} \to R^{d^{'}}$ ⁠, let $\partial_{k} ϕ : R^{d} \to R$ be the partial derivative $\partial_{k} ϕ$ with respect to the $k$ th variable of $ϕ$ ⁠. For the syntax of any differentiation map $\partial$ ⁠, given functions $ϕ$ and $ζ$ ⁠, let $\partial ϕ (ζ (q)) = (\partial ϕ) (ζ (q))$ be the (partial) derivative $\partial ϕ$ evaluated at an output $ζ (q)$ of a function $ζ$ ⁠.

Given a matrix $M \in R^{d \times d^{'}}$ ⁠, $vec (M) = {[M_{1, 1}, \dots, M_{d, 1}, M_{1, 2}, \dots, M_{d, 2}, \dots, M_{1, d^{'}}, \dots, M_{d, d^{'}}]}^{T}$ represents the standard vectorization of the matrix $M$ ⁠. Given a set of $n$ matrices or vectors ${M^{(j)}}_{j = 1}^{n}$ ⁠, define ${[M^{(j)}]}_{j = 1}^{n} = [M^{(1)}, M^{(2)}, \dots, M^{(n)}]$ to be a block matrix of each column block being $M^{(1)}, M^{(2)}, \dots, M^{(n)}$ ⁠. Similarly, given a set $I = {i_{1}, \dots, i_{n}}$ with $(i_{1}, \dots, i_{n})$ increasing, define ${[M^{(j)}]}_{j \in I} = [\begin{matrix} M^{(i_{1})} & \dots & M^{(i_{n})} \end{matrix}]$ ⁠.

3 Nonconvex Optimization Landscapes for Machine Learning

This section shows our first main result that under mild assumptions, every differentiable local minimum achieves the global optimality of the perturbable gradient basis model class.

3.1 Assumptions

Given a hypothesis class $f$ and data set, let $Ω$ be a set of nondifferentiable points $θ$ as $Ω = {θ \in R^{d_{θ}} : (\exists i \in {1, \dots, m}) [f_{x_{i}}$ is not differentiable at $θ]}$ ⁠. Similarly, define $\tilde{Ω} = {θ \in R^{d_{θ}} : (\forall ε > 0) (\exists θ^{'} \in B (θ, ε)) (\exists i \in {1, \dots, m}) [f_{x_{i}}$ is not differentiable at $θ^{'}]}$ ⁠. Here, $B (θ, ε)$ is the open ball with the center $θ$ and the radius $ε$ ⁠. In common nondifferentiable models $f$ such as neural networks with rectified linear units (ReLUs) and pooling operations, we have that $Ω = \tilde{Ω}$ ⁠, and the Lebesgue measure of $Ω (= \tilde{Ω}$ ⁠) is zero.

This section uses the following mild assumptions.

Assumption 1

(Use of Common Loss criteria). For all $i \in {1, \dots, m}$ ⁠, the function $ℓ_{y_{i}} : q \mapsto ℓ (q, y_{i}) \in R_{\geq 0}$ is differentiable and convex (e.g., the squared loss, cross-entropy loss, or polynomial hinge loss satisfies this assumption).

Assumption 2

(Use of Common Model Structures). There exists a function $g : R^{d_{θ}} \to R^{d_{θ}}$ such that $f_{x_{i}} (θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ)$ for all $i \in {1, \dots, m}$ and all $θ \in R^{d_{θ}} ∖ Ω$ ⁠.

Assumption 1 is satisfied by simply using common loss criteria that include the squared loss $ℓ (q, y) = {∥ q - y ∥}_{2}^{2}$ ⁠, cross-entropy loss $ℓ (q, y) = - \sum_{k = 1}^{d_{y}} y_{k} log \frac{exp (q_{k})}{\sum_{k^{'}} exp (q_{k^{'}})}$ ⁠, and smoothed hinge loss $ℓ (q, y) = {(max {0, 1 - y q})}^{p}$ with $p \geq 2$ (the hinge loss with $d_{y} = 1$ ⁠). Although the objective function $L : θ \mapsto L (θ)$ used to train a complex machine learning model (e.g., a neural network) is nonconvex in $θ$ ⁠, the loss criterion $ℓ_{y_{i}} : q \mapsto ℓ (q, y_{i})$ is usually convex in $q$ ⁠. In this article, the cross-entropy loss includes the softmax function, and thus $f_{x} (θ)$ is the pre-softmax output of the last layer in related deep learning models.

Assumption 2 is satisfied by simply using a common architecture in deep learning or a classical machine learning model. For example, consider a deep neural network of the form $f_{x} (θ) = W h (x; u) + b$ ⁠, where $h (x; u)$ is an output of an arbitrary representation at the last hidden layer and $θ = vec ([W, b, u])$ ⁠. Then assumption 2 holds because $f_{x_{i}} (θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ)$ ⁠, where $g {(θ)}_{k} = θ_{k}$ for all $k$ corresponding to the parameters $(W, b)$ in the last layer and $g {(θ)}_{k} = 0$ for all other $k$ corresponding to $u$ ⁠. In general, because $g$ is a function of $θ$ ⁠, assumption 2 is easily satisfiable. Assumption 2 does not require the model $f (x; θ)$ to be linear in $θ$ or $x$ ⁠.

Note that we allow the nondifferentiable points to exist in $L (θ)$ ⁠; for example, the use of ReLU is allowed. For a nonconvex and nondifferentiable function, we can still have first-order and second-order necessary conditions of local minima (e.g., Rockafellar & Wets, 2009, theorem 13.24). However, subdifferential calculus of a nonconvex function requires careful treatment at nondifferentiable points (see Rockafellar & Wets, 2009; Kakade & Lee, 2018; Davis, Drusvyatskiy, Kakade, & Lee, 2019), and deriving guarantees at nondifferentiable points is left to a future study.

3.2 Theory for Critical Points

Before presenting the first main result, this section provides a simpler result for critical points to illustrate the ideas behind the main result for local minima. We define the (theoretical) objective function

L_{θ}

of the gradient basis model class as

L_{θ} (α) = \sum_{i = 1}^{m} λ_{i} ℓ (f_{θ} (x_{i}; α), y_{i}),

where

{f_{θ} (x_{i}; α) = \sum_{k = 1}^{d_{θ}} α_{k} \partial_{k} f_{x_{i}} (θ) : α \in R^{d_{θ}}}

is the induced gradient basis model class. The following theorem shows that every differentiable critical point of our original objective

L

(including every differentiable local minimum and saddle point) achieves the global minimum value of

L_{θ}

⁠. The complete proofs of all the theoretical results are presented in appendix A.

Theorem 1.

Let assumptions 1 and 2 hold. Then for any critical point

θ \in (R^{d_{θ}} ∖ Ω)

of

L

⁠, the following holds:

L (θ) = inf_{α \in R^{d_{θ}}} L_{θ} (α) .

An important aspect in theorem 1 is that $L_{θ}$ on the right-hand side is convex, while $L$ on the left-hand side can be nonconvex or convex. Here, following convention, $inf S$ is defined to be the infimum of a subset $S$ of $\bar{R}$ (the set of affinely extended real numbers); that is, if $S$ has no lower bound, $inf S = - \infty$ and $inf \emptyset = \infty$ ⁠. Note that theorem 1 vacuously holds true if there is no critical point for $L$ ⁠. To guarantee the existence of a minimizer in a (nonempty) subspace $S \subseteq R^{d_{θ}}$ for $L$ (or $L_{θ}$ ⁠), a classical proof requires two conditions: a lower semicontinuity of $L$ (or $L_{θ}$ ⁠) and the existence of a $q \in S$ for which the set ${q^{'} \in S : L (q^{'}) \leq L (q)}$ (or ${q^{'} \in S : L_{θ} (q^{'}) \leq L_{θ} (q)}$ ⁠) is compact (see Bertsekas, 1999, for different conditions).

3.2.1 Geometric View

This section presents the geometric interpretation of theorem 1 that provides an intuitive yet formal description of gradient basis model class. Figure 1 illustrates the gradient basis model class and theorem 1 with

θ \in R^{2}

and

f_{X} (θ) \in R^{3}

⁠. Here, we consider the following map from the parameter space to the concatenation of the output of the model at

x_{1}, x_{2}, \dots, x_{m}

⁠:

\begin{matrix} f_{X} : θ \in R^{d_{θ}} \mapsto {(f_{x_{1}} {(θ)}^{⊤}, f_{x_{2}} {(θ)}^{⊤}, \dots, f_{x_{m}} {(θ)}^{⊤})}^{⊤} \in R^{m d_{y}} . \end{matrix}

Figure 1:

Illustration of gradient basis model class and theorem 1 with θ∈R2 and fX(θ)∈R3 (dy=1). Theorem 1 translates the local condition of θ in the parameter space R2 (on the left) to the global optimality in the output space R3 (on the right). The subspace TfX(θ) is the space of the outputs of the gradient basis model class. Theorem 1 states that fX(θ) is globally optimal in the subspace as fX(θ)∈argminf∈TfX(θ)dist(f,y) for any differentiable critical point θ of L.

View large Download slide

Illustration of gradient basis model class and theorem 1 with $θ \in R^{2}$ and $f_{X} (θ) \in R^{3}$ (⁠ $d_{y} = 1$ ⁠). Theorem 1 translates the local condition of $θ$ in the parameter space $R^{2}$ (on the left) to the global optimality in the output space $R^{3}$ (on the right). The subspace $T_{f_{X} (θ)}$ is the space of the outputs of the gradient basis model class. Theorem 1 states that $f_{X} (θ)$ is globally optimal in the subspace as $f_{X} (θ) \in {argmin}_{f \in T_{f_{X} (θ)}} dist (f, y)$ for any differentiable critical point $θ$ of $L$ ⁠.

In the output space

R^{m d_{y}}

of

f_{X}

⁠, the objective function

L

induces the notion of distance from the target vector

y = {(y_{1}^{⊤}, \dots, y_{m}^{⊤})}^{⊤} \in R^{m d_{y}}

to a vector

f = {(f_{1}^{⊤}, \dots, f_{m}^{⊤})}^{⊤} \in R^{m d_{y}}

as

dist (f, y) = \sum_{i = 1}^{m} λ_{i} ℓ (f_{i}, y_{i}) .

We consider the affine subspace

T_{f_{X} (θ)}

of

R^{m d_{y}}

that passes through the point

f_{X} (θ)

and is spanned by the set of vectors

{\partial_{1} f_{X} (θ), \dots, \partial_{d_{θ}} f_{X} (θ)}

⁠,

T_{f_{X} (θ)} = span ({\partial_{1} f_{X} (θ), \dots, \partial_{d_{θ}} f_{X} (θ)}) + {f_{X} (θ)},

where the sum of the two sets represents the Minkowski sum of the sets.

Then the subspace

T_{f_{X} (θ)}

is the space of the outputs of the gradient basis model class in general beyond the low-dimensional illustration. This is because by assumption 2, for any given

θ

⁠,

\begin{matrix} T_{f_{X} (θ)} & = & \{\sum_{k = 1}^{d_{θ}} (g {(θ)}_{k} + α_{k}) \partial_{k} f_{X} (θ) : α \in R^{d_{θ}}\} \\ = & \{\sum_{k = 1}^{d_{θ}} α_{k} \partial_{k} f_{X} (θ) : α \in R^{d_{θ}}\}, \end{matrix}

(3.1)

and

\sum_{k = 1}^{d_{θ}} α_{k} \partial_{k} f_{X} (θ) = {(f_{θ} {(x_{1}; α)}^{⊤}, \dots, f_{θ} {(x_{m}; α)}^{⊤})}^{⊤}

⁠. In other words,

T_{f_{X} (θ)} = span ({\partial_{1} f_{X} (θ), \dots, \partial_{d_{θ}} f_{X} (θ)}) ∋ {(f_{θ} {(x_{1}; α)}^{⊤}, \dots, f_{θ} {(x_{m}; α)}^{⊤})}^{⊤}

⁠.

Therefore, in general, theorem 1 states that under assumptions 1 and 2,

f_{X} (θ)

is globally optimal in the subspace

T_{f_{X} (θ)}

as

f_{X} (θ) \in \underset{f \in T_{f_{X} (θ)}}{argmin} dist (f, y),

for any differentiable critical point

θ

of

L

⁠. Theorem 1 concludes this global optimality in the affine subspace of the output space based on the local condition in the parameter space (i.e., differentiable critical point). A key idea behind theorem 1 is to consider the map between the parameter space and the output space, which enables us to take advantage of assumptions 1 and 2.

Figure 2 illustrates the gradient basis model class and theorem 1 with a union of manifolds and a tangent space. Under the constant rank condition, the image of the map $f_{X}$ locally forms a single manifold. More precisely, if there exists a small neighborhood $U (θ)$ of $θ$ such that $f_{X}$ is differentiable in $U (θ)$ and $rank (\partial f_{X} (θ^{'})) = r$ is constant with some $r$ for all $θ^{'} \in U (θ)$ (the constant rank condition), then the rank theorem states that the image $f_{X} (U (θ))$ is a manifold of dimension $r$ (Lee, 2013, theorem 4.12). We note that the rank map $θ \mapsto rank (\partial f_{X} (θ))$ is lower semicontinuous (i.e., if $rank (\partial f_{X} (θ)) = r$ ⁠, then there exists a neighborhood $U (θ)$ of $θ$ such that $rank (\partial f_{X} (θ^{'})) \geq r$ for any $θ^{'} \in U (θ)$ ⁠). Therefore, if $\partial f_{X} (θ)$ at $θ$ has the maximum rank in a small neighborhood of $θ$ ⁠, then the constant rank condition is satisfied.

Figure 2:

View large Download slide

Illustration of gradient basis model class and theorem 1 with manifold and tangent space. The space $R^{2} ∋ θ$ on the left is the parameter space, and the space $R^{3} ∋ f_{X} (θ)$ on the right is the output space. The surface $M \subset R^{3}$ on the right is the image of $f_{X}$ ⁠, which is a union of finitely many manifolds. The tangent space $T_{f_{X} (θ)}$ is the space of the outputs of the gradient basis model class. Theorem 1 states that if $θ$ is a differentiable critical point of $L$ ⁠, then $f_{X} (θ)$ is globally optimal in the tangent space $T_{f_{X} (θ)}$ ⁠.

For points

θ

where the constant rank condition is violated, the image of the map

f_{X}

is no longer a single manifold. However, locally it decomposes as a union of finitely many manifolds. More precisely, if there exists a small neighborhood

U (θ)

of

θ

such that

f_{X}

is analytic over

U (θ)

(this condition is satisfied for commonly used activation functions such as ReLU, sigmoid, and hyperbolic tangent at any differentiable point), then the image

f_{X} (U (θ))

admits a locally finite partition

M

into connected submanifolds such that whenever

M \neq M^{'} \in M

with

\bar{M} \cap M^{'} \neq \emptyset

(⁠

\bar{M}

is the closure of

M

⁠), we have

\begin{matrix} M^{'} \subset \bar{M}, dim (M^{'}) < dim (M) . \end{matrix}

See Hardt (1975) for the proof.

If the point

θ

satisfies the constant rank condition, then

T_{f_{X} (θ)}

is exactly the tangent space of the manifold formed by the image

f_{X} (U (θ))

⁠. Otherwise, locally the image decomposes into a finite union

M

of submanifolds. In this case,

T_{f_{X} (θ)}

belongs to the span of the tangent space of those manifolds in

M

as

\begin{matrix} T_{f_{X} (θ)} \subset {T_{p} M : p = f_{X} (θ), M \in M}, \end{matrix}

where

T_{p} M

is the tangent space of the manifold

M

at the point

p

⁠.

3.2.2 Examples

In this section, we show through examples that theorem 1 generalizes the previous results in special cases while providing new theoretical insights based on the gradient basis model class and its geometric view. In the following, whenever the form of $f$ is specified, we require only assumption 1 because assumption 2 is automatically satisfied by a given $f$ ⁠.

For classical machine learning models, example 1 shows that the gradient basis model class is indeed equivalent to a given model class. From the geometric view, this means that for any $θ$ ⁠, the tangent space $T_{f_{X} (θ)}$ is equal to the whole image $M$ of $f_{X}$ (i.e., $T_{f_{X} (θ)}$ does not depend on $θ$ ⁠). This reduces theorem 1 to the statement that every critical point of $L$ is a global minimum of $L$ ⁠.

Example 1: Classical Machine Learning Models.

For any basis function model $f (x; θ) = \sum_{k = 1}^{d_{θ}} θ_{k} φ {(x)}_{k}$ in classical machine learning with any fixed feature map $φ : X \to R^{d_{θ}}$ ⁠, we have that $f_{θ} (x; α) = f (x; α)$ ⁠, and hence ${inf}_{θ \in R^{d_{θ}}} L (θ) = {inf}_{α \in R^{d_{θ}}} L_{θ} (α)$ ⁠, as well as $Ω = \emptyset$ ⁠. In other words, in this special case, theorem 1 states that every critical point of $L$ is a global minimum of $L$ ⁠. Here, we do not assume that a critical point or a global minimum exists or can be attainable. Instead, the statement logically means that if a point is a critical point, then the point is a global minimum. This type of statement vacuously holds true if there is no critical point.

For overparameterized deep neural networks, example 2 shows that the induced gradient basis model class is highly expressive such that it must contain the globally optimal model of a given model class of deep neural networks. In this example, the tangent space $T_{f_{X} (θ)}$ is equal to the whole output space $R^{m d_{y}}$ ⁠. This reduces theorem 1 to the statement that every critical point of $L$ is a global minimum of $L$ for overparameterized deep neural networks.

Intuitively, in Figure 1 or 2, we can increase the number of parameters and raise the number of partial derivatives $\partial_{k} f_{X} (θ)$ in order to increase the dimensionality of the tangent space $T_{f_{X} (θ)}$ so that $T_{f_{X} (θ)} = R^{m d_{y}}$ ⁠. This is indeed what happens in example 2, as well as in the previous studies of significantly overparameterized deep neural networks (Allen-Zhu, Li, & Song, 2018; Du, Lee, Li, Wang, & Zhai, 2018; Zou et al., 2018). In the previous studies, the significant overparameterization is required so that the tangent space $T_{f_{X} (θ)}$ does not change from the initial tangent space $T_{f_{X} (θ^{(0)})} = R^{m d_{y}}$ during training. Thus, theorem 1, with its geometric view, provides the novel algebraic and geometric insights into the results of the previous studies and the reason why overparameterized deep neural networks are easy to be optimized despite nonconvexity.

Example 2: Overparameterized Deep Neural Networks.

Theorem 1 implies that every critical point (and every local minimum) is a global minimum for sufficiently overparameterized deep neural networks. Let $n$ be the number of units in each layer of a fully connected feedforward deep neural network. Let us consider a significant overparameterization such that $n \geq m$ ⁠. Let us write a fully connected feedforward deep neural network with the trainable parameters $(θ, u)$ by $f (x; θ) = W φ (x; u)$ ⁠, where $W \in R^{d_{y} \times n}$ is the weight matrix in the last layer, $θ = vec (W)$ ⁠, $u$ contains the rest of the parameters, and $φ (x; u)$ is the output of the last hidden layer. Denote $x_{i} = {[{(x_{i}^{(raw)})}^{⊤}, 1]}^{⊤}$ to contain the constant term to account for the bias term in the first layer. Assume that the input samples are normalized as $∥ x_{i}^{(raw)} ∥_{2} = 1$ for all $i \in {1, \dots, m}$ and distinct as ${(x_{i}^{(raw)})}^{⊤} x_{i^{'}}^{(raw)} < 1 - δ$ with some $δ > 0$ for all $i^{'} \neq i$ ⁠. Assume that the activation functions are ReLU activation functions. Then we can efficiently set $u$ to guarantee $rank ({[φ (x_{i}; u)]}_{i = 1}^{m}) \geq m$ (e.g., by choosing $u$ to make each unit of the last layer to be active only for each sample $x_{i}$ ⁠).^¹ Theorem 1 implies that every critical point $θ$ with this $u$ is a global minimum of the whole set of trainable parameters $(θ, u)$ because ${inf}_{α} L_{θ} (α) = {inf}_{f_{1}, \dots, f_{m}} \sum_{i = 1}^{m} λ_{i} ℓ (f_{i}, y_{i})$ (with assumption 1).

For deep neural networks, example 3 shows that standard networks have the global optimality guarantee with respect to the representation learned at the last layer, and skip connections further ensure the global optimality with respect to the representation learned at each hidden layer. This is because adding the skip connections incurs new partial derivatives ${\partial_{k} f_{X} (θ)}_{k}$ that span the tangent space containing the output of the best model with the corresponding learned representation.

Example 3: Deep Neural Networks and Learned Representations.

Consider a feedforward deep neural network, and let

I^{(skip)} \subseteq {1, \dots, H}

be the set of indices such that there exists a skip connection from the

(l - 1)

th layer to the last layer for all

l \in I^{(skip)}

⁠; that is, in this example,

f (x; θ) = \sum_{l \in I^{(skip)}} W^{(l + 1)} h^{(l)} (x; u),

where

θ = vec ([{[W^{(l + 1)}]}_{l \in I^{(skip)}}, u]) \in R^{d_{θ}}

with

W^{(l + 1)} \in R^{d_{y} \times d_{l}}

and

u \in R^{d_{u}}

⁠.

The conclusion in this example holds for standard deep neural networks without skip connections too, since we always have

H \in I^{(skip)}

for standard deep neural networks. Let assumption 1 hold. Then theorem 1 implies that for any critical point

θ \in (R^{d_{θ}} ∖ Ω)

of

L

⁠, the following holds:

L (θ) = inf_{α \in R^{d_{θ}}} L_{θ}^{(skip)} (α),

where

\begin{matrix} L_{θ}^{(skip)} (α) = \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (\sum_{l \in I^{(skip)}} α_{w}^{(l + 1)} h^{(l)} (x_{i}; u) + \sum_{k = 1}^{d_{u}} {(α_{u})}_{k} \partial_{u_{k}} f_{x_{i}} (θ)), \end{matrix}

with

α = vec ([{[α^{(l + 1)}]}_{l \in I^{(skip)}}, α_{u}]) \in R^{d_{θ}}

with

α^{(l + 1)} \in R^{d_{y} \times d_{l}}

and

α_{u} \in R^{d_{u}}

⁠. This is because

f (x; θ) = (\partial_{vec (W^{(H + 1)})} f (x; θ)) vec (W^{(H + 1)})

⁠, and thus assumption 2 is automatically satisfied. Here,

h^{(l)} (x_{i}; u)

is the representation learned at the

l

-layer. Therefore,

{inf}_{α \in R^{d_{θ}}} L_{θ}^{(skip)} (α)

is at most the global minimum value of the basis models with the learned representations of the last layer and all hidden layers with the skip connections.

3.3 Theory for Local Minima

We are now ready to present our first main result. We define the (theoretical) objective function

{\tilde{L}}_{θ}

of the perturbable gradient basis model class as

{\tilde{L}}_{θ} (α, ε, S) = \sum_{i = 1}^{m} λ_{i} ℓ ({\tilde{f}}_{θ} (x_{i}; α, ε, S), y_{i}),

where

{\tilde{f}}_{θ} (x_{i}; α, ε, S)

is a perturbed gradient basis model defined as

{\tilde{f}}_{θ} (x_{i}; α, ε, S) = \sum_{k = 1}^{d_{θ}} \sum_{j = 1}^{| S |} α_{k, j} \partial_{k} f_{x_{i}} (θ + ε S_{j}) .

Here,

S

is a finite set of vectors

S_{1}, \dots, S_{| S |} \in R^{d_{θ}}

and

α \in R^{d_{θ} \times | S |}

⁠. Let

V [θ, ε]

be the set of all vectors

v \in R^{d_{θ}}

such that

{∥ v ∥}_{2} \leq 1

and

f_{x_{i}} (θ + ε v) = f_{x_{i}} (θ)

for any

i \in {1, \dots, m}

⁠. Let

S \subseteq_{fin} S^{'}

denote a finite subset

S

of a set

S^{'}

⁠. For an

S_{j} \in V [θ, ε]

⁠, we have

f_{x_{i}} (θ + ε S_{j}) = f_{x_{i}} (θ)

⁠, but it is possible to have

\partial_{k} f_{x_{i}} (θ + ε S_{j}) \neq \partial_{k} f_{x_{i}} (θ)

⁠. This enables the greater expressivity of

{\tilde{f}}_{θ} (x_{i}; α, ε, S)

with a

S \subseteq_{fin} V [θ, ε]

when compared with

f_{θ} (x_{i}; α)

⁠.

The following theorem shows that every differentiable local minimum of $L$ achieves the global minimum value of ${\tilde{L}}_{θ}$ ⁠:

Theorem 2.

Let assumptions 1 and 2 hold. Then, for any local minimum

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

of

L

⁠, the following holds: there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠,

\begin{matrix} L (θ) = inf_{\binom{S \subseteq_{f i n} V [θ, ε],}{α \in R^{d_{θ} \times | S |}}} {\tilde{L}}_{θ} (α, ε, S) . \end{matrix}

(3.2)

To understand the relationship between theorems 1 and 2, let us consider the following general inequalities: for any

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

with

ε \geq 0

being sufficiently small,

L (θ) \geq inf_{α \in R^{d_{θ}}} L_{θ} (α) \geq inf_{\binom{S \subseteq_{f i n} V [θ, ε_{}],}{α \in R^{d_{θ} \times | S |}}} {\tilde{L}}_{θ} (α, ε, S) .

Here, whereas theorem 1 states that the first inequality becomes equality as

L (θ) = {inf}_{α \in R^{d_{θ}}} L_{θ} (α)

at every differentiable critical point, theorem 2 states that both inequalities become equality as

L (θ) = inf_{α \in R^{d_{θ}}} L_{θ} (α) = inf_{\binom{S \subseteq_{f i n} V [θ, ε],}{α \in R^{d_{θ} \times | S |}}} {\tilde{L}}_{θ} (α, ε, S)

at every differentiable local minimum.

From theorem 1 to theorem 2, the power of increasing the number of parameters (including overparameterization) is further improved. The right-hand side in equation 3.2 is the global minimum value over the variables $S \subseteq_{fin} V [θ, ε]$ and $α \in R^{d_{θ} \times | S |}$ ⁠. Here, as $d_{θ}$ increases, we may obtain the global minimum value of a larger search space $R^{d_{θ} \times | S |}$ ⁠, which is similar to theorem 1. A concern in theorem 1 is that as $d_{θ}$ increases, we may also significantly increase the redundancy among the elements in ${\partial_{k} f_{x_{i}} (θ)}_{k = 1}^{d_{θ}}$ ⁠. Although this remains a valid concern, theorem 2 allows us to break the redundancy by the globally optimal $S \subseteq_{fin} V [θ, ε]$ to some degree.

For example, consider $f (x; θ) = g (W^{(l)} h^{(l)} (x; u); u)$ ⁠, which represents a deep neural network, with some $l$ th-layer output $h^{(l)} (x; u) \in R^{d_{l}}$ ⁠, a trainable weight matrix $W^{(l)}$ ⁠, and an arbitrary function $g$ to compute the rest of the forward pass. Here, $θ = vec ([W^{(l)}, u])$ ⁠. Let $h^{(l)} (X; u) = {[h^{(l)} (x_{i}; u)]}_{i = 1}^{m} \in R^{d_{l} \times m}$ and, similarly, $f (X; θ) = g (W^{(l)} h^{(l)} (X; u); u) \in R^{d_{y} \times m}$ ⁠. Then, all vectors $v$ corresponding to any elements in the left null space of $h^{(l)} (X; u)$ are in $V [θ, ε]$ (i.e., $v_{k} = 0$ for all $k$ corresponding to $u$ and the rest of $v_{k}$ is set to perturb $W^{(l)}$ by an element in the left null space). Thus, as the redundancy increases such that the dimension of the left null space of $h^{(l)} (X; u)$ increases, we have a larger space of $V [θ, ε]$ ⁠, for which a global minimum value is guaranteed at a local minimum.

3.3.1 Geometric View

This section presents the geometric interpretation of the perturbable gradient basis model class and theorem 2. Figure 3 illustrates the perturbable gradient basis model class and theorem 2 with

θ \in R^{2}

and

f_{X} (θ) \in R^{3}

⁠. Figure 4 illustrates them with a union of manifolds and tangent spaces at a singular point. Given a

ε

(⁠

\leq ε_{0}

⁠), define the affine subspace

{\tilde{T}}_{f_{X} (θ)}

of the output space

R^{m d_{y}}

by

{\tilde{T}}_{f_{X} (θ)} = span ({f \in R^{m d_{y}} : (\exists v \in V [θ, ε]) [f \in T_{f_{X} (θ + ε v)}]}) .

Then the subspace

{\tilde{T}}_{f_{X} (θ)}

is the space of the outputs of the perturbable gradient basis model class in general beyond the low-dimensional illustration (this follows equation and the definition of the perturbable gradient basis model). Therefore, in general, theorem 2 states that under assumptions 1 and 2,

f_{X} (θ)

is globally optimal in the subspace

{\tilde{T}}_{f_{X} (θ)}

as

f_{X} (θ) \in \underset{f \in {\tilde{T}}_{f_{X} (θ)}}{argmin} dist (f, y)

for any differentiable local minima

θ

of

L

⁠. Theorem 2 concludes the global optimality in the affine subspace of the output space based on the local condition in the parameter space—that is, differentiable local minima. Here, a (differentiable) local minimum

θ

is required to be optimal only in an arbitrarily small local neighborhood in the parameter space, and yet

f_{X} (θ)

is guaranteed to be globally optimal in the affine subspace of the output space. This illuminates the fact that nonconvex optimization in machine learning has a particular structure beyond general nonconvex optimization.

Figure 3:

Illustration of perturbable gradient basis model class and theorem 2 with θ∈R2 and fX(θ)∈R3 (dy=1). Theorem 2 translates the local condition of θ in the parameter space R2 (on the left) to the global optimality in the output space R3 (on the right). The subspace T˜fX(θ) is the space of the outputs of the perturbable gradient basis model class. Theorem 2 states that fX(θ) is globally optimal in the subspace as fX(θ)∈argminf∈T˜fX(θ)dist(f,y) for any differentiable local minima θ of L. In this example, T˜fX(θ) is the whole output space R3, while TfX(θ) is not, illustrating the advantage of the perturbable gradient basis over the gradient basis. Since T˜fX(θ)=R3, fX(θ) must be globally optimal in the whole output space R3.

View large Download slide

Illustration of perturbable gradient basis model class and theorem 2 with $θ \in R^{2}$ and $f_{X} (θ) \in R^{3}$ (⁠ $d_{y} = 1$ ⁠). Theorem 2 translates the local condition of $θ$ in the parameter space $R^{2}$ (on the left) to the global optimality in the output space $R^{3}$ (on the right). The subspace ${\tilde{T}}_{f_{X} (θ)}$ is the space of the outputs of the perturbable gradient basis model class. Theorem 2 states that $f_{X} (θ)$ is globally optimal in the subspace as $f_{X} (θ) \in {argmin}_{f \in {\tilde{T}}_{f_{X} (θ)}} dist (f, y)$ for any differentiable local minima $θ$ of $L$ ⁠. In this example, ${\tilde{T}}_{f_{X} (θ)}$ is the whole output space $R^{3}$ ⁠, while $T_{f_{X} (θ)}$ is not, illustrating the advantage of the perturbable gradient basis over the gradient basis. Since ${\tilde{T}}_{f_{X} (θ)} = R^{3}$ ⁠, $f_{X} (θ)$ must be globally optimal in the whole output space $R^{3}$ ⁠.

Figure 4:

Illustration of perturbable gradient basis model class and theorem 2 with manifold and tangent space at a singular point. The surface M⊂R3 is the image of fX, which is a union of finitely many manifolds. The line TfX(θ) on the left panel is the space of the outputs of the gradient basis model class. The whole space T˜fX(θ)=R3 on the right panel is the space of the outputs of the perturbable gradient basis model class. The space T˜fX(θ) is the span of the set of the vectors in the tangent spaces TfX(θ),TfX(θ'), and TfX(θ''). Theorem 2 states that if θ is a differentiable local minimum of L, then fX(θ) is globally optimal in the space T˜fX(θ).

View large Download slide

Illustration of perturbable gradient basis model class and theorem 2 with manifold and tangent space at a singular point. The surface $M \subset R^{3}$ is the image of $f_{X}$ ⁠, which is a union of finitely many manifolds. The line $T_{f_{X} (θ)}$ on the left panel is the space of the outputs of the gradient basis model class. The whole space ${\tilde{T}}_{f_{X} (θ)} = R^{3}$ on the right panel is the space of the outputs of the perturbable gradient basis model class. The space ${\tilde{T}}_{f_{X} (θ)}$ is the span of the set of the vectors in the tangent spaces $T_{f_{X} (θ)}, T_{f_{X} (θ^{'})}$ ⁠, and $T_{f_{X} (θ^{''})}$ ⁠. Theorem 2 states that if $θ$ is a differentiable local minimum of $L$ ⁠, then $f_{X} (θ)$ is globally optimal in the space ${\tilde{T}}_{f_{X} (θ)}$ ⁠.

4 Applications to Deep Neural Networks

The previous section showed that all local minima achieve the global optimality of the perturbable gradient basis model class with several direct consequences for special cases. In this section, as consequences of theorem 2, we complement or improve the state-of-the-art results in the literature.

4.1 Example: ResNets

As an example of theorem 2, we set

f

to be the function of a certain type of residual networks (ResNets) that Shamir (2018) studied. That is, both Shamir (2018) and this section set

f

as

\begin{matrix} f (x; θ) = W (x + R z (x; u)), \end{matrix}

(4.1)

where

θ = vec ([W, R, u]) \in R^{d_{θ}}

with

W \in R^{d_{y} \times d_{x}}

⁠,

R \in R^{d_{x} \times d_{z}}

⁠, and

u \in R^{d_{u}}

⁠. Here,

z (x; u) \in R^{d_{z}}

represents an output of deep residual functions with a parameter vector

u

⁠. No assumption is imposed on the form of

z (x; u)

⁠, and

z (x; u)

can represent an output of possibly complicated deep residual functions that arise in ResNets. For example, the function

f

can represent deep preactivation ResNets (He, Zhang, Ren, & Sun, 2016), which are widely used in practice. To simplify theoretical study, Shamir (2018) assumed that every entry of the matrix

R

is unconstrained (e.g., instead of

R

representing convolutions). We adopt this assumption based on the previous study (Shamir, 2018).

4.1.1 Background

Along with an analysis of approximate critical points, Shamir (2018) proved the following main result, proposition 1, under the assumptions PA1, PA2, and PA3:

PA1: The output dimension $d_{y} = 1$ ⁠.
PA2: For any $y$ ⁠, the function $ℓ_{y}$ is convex and twice differentiable.
PA3: On any bounded subset of the domain of $L$ ⁠, the function $L_{u} (W, R)$ ⁠, its gradient $\nabla L_{u} (W, R)$ ⁠, and its Hessian $\nabla^{2} L_{u} (W, R)$ are all Lipschitz continuous in $(W, R)$ ⁠, where $L_{u} (W, R) = L (θ)$ with a fixed $u$ ⁠.

Proposition 1

(Shamir, 2018). Let

f

be specified by equation , Let assumptions PA1, PA2, and PA3 hold. Then for any local minimum

θ

of

L

⁠,

L (θ) \leq inf_{W \in R^{d_{y} \times d_{x}}} \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (W x_{i}) .

Shamir (2018) remarked that it is an open problem whether proposition 1 and another main result in the article can be extended to networks with $d_{y} > 1$ (multiple output units). Note that Shamir (2018) also provided proposition 1 with an expected loss and an analysis for a simpler decoupled model, $W x + V z (x; u)$ ⁠. For the simpler decoupled model, our theorem 1 immediately concludes that given any $u$ ⁠, every critical point with respect to $θ_{- u} = (W, R)$ achieves a global minimum value with respect to $θ_{- u}$ as $L (θ_{- u}) = inf {\sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (W x_{i} + R z (x_{i}; u)) : W \in R^{d_{y} \times d_{x}}, R \in R^{d_{x} \times d_{z}}}$ (⁠ $\leq {inf}_{W \in R^{d_{y} \times d_{x}}} \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (W x_{i})$ ⁠). This holds for every critical point $θ$ since any critical point $θ$ must be a critical point with respect to $θ_{- u}$ ⁠.

4.2 Result

The following theorem shows that every differentiable local minimum achieves the global minimum value of ${\tilde{L}}_{θ}^{(ResNet)}$ (the right-hand side in equation 4.2), which is no worse than the upper bound in proposition 1 and is strictly better than the upper bound as long as $z (x_{i}, u)$ or ${\tilde{f}}_{θ} (x_{i}; α, ε, S)$ is nonnegligible. Indeed, the global minimum value of ${\tilde{L}}_{θ}^{(ResNet)}$ (the right-hand side in equation 4.2) is no worse than the global minimum value of all models parameterized by the coefficients of the basis $x$ and $z (x; u)$ ⁠, and further improvement is guaranteed through a nonnegligible ${\tilde{f}}_{θ} (x_{i}; α, ε, S)$ ⁠.

Theorem 3.

Let

f

be specified by equation . Let assumption 1 hold. Assume that

d_{y} \leq min {d_{x}, d_{z}}

⁠. Then for any local minimum

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

of

L

⁠, the following holds: there exists

ε_{0} > 0

such that for any

ε \in (0, ε_{0})

⁠,

\begin{matrix} L (θ) = inf_{\begin{matrix} S \subseteq_{f i n} V [θ, ε], \\ α \in R^{d_{θ} \times | S |}, \\ α_{w} \in R^{d_{y} \times d_{x}}, α_{r} \in R^{d_{y} \times d_{z}} \end{matrix}} {\tilde{L}}_{θ}^{(R e s N e t)} (α, α_{w}, α_{r}, ε, S), \end{matrix}

(4.2)

where

\begin{matrix} {\tilde{L}}_{θ}^{(R e s N e t)} (α, α_{w}, α_{r}, ε, S) = \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (α_{w} x_{i} + α_{r} z (x_{i}; u) + {\tilde{f}}_{θ} (x_{i}; α, ε, S)) . \end{matrix}

Theorem 3 also successfully solved the first part of the open problem in the literature (Shamir, 2018) by discarding the assumption of $d_{y} = 1$ ⁠. From the geometric view, theorem 3 states that the span ${\tilde{T}}_{f_{X} (θ)}$ of the set of the vectors in the tangent spaces ${T_{f_{X} (θ + ε v)} : v \in V [θ, ε]}$ contains the output of the best basis model with the linear feature $x$ and the learned nonlinear feature $z (x_{i}; u)$ ⁠. Similar to the examples in Figures 3 and 4, ${\tilde{T}}_{f_{X} (θ)} \neq T_{f (θ)}$ and the output of the best basis model with these features is contained in ${\tilde{T}}_{f_{X} (θ)}$ but not in $T_{f (θ)}$ ⁠.

Unlike the recent study on ResNets (Kawaguchi & Bengio, 2019), our theorem 3 predicts the value of $L$ through the global minimum value of a large search space (i.e., the domain of ${\tilde{L}}_{θ}^{(ResNet)}$ ⁠) and is proven as a consequence of our general theory (i.e., theorem 2) with a significantly different proof idea (see section 4.3) and with the novel geometric insight.

4.2.1 Example: Deep Nonlinear Networks with Locally Induced Partial Linear Structures

We specify

f

to represent fully connected feedforward networks with arbitrary nonlinearity

σ

and arbitrary depth

H

as follows:

\begin{matrix} f (x; θ) = W_{}^{(H + 1)} h^{(H)} (x; θ), \end{matrix}

(4.3)

where

h^{(l)} (x; θ) = σ^{(l)} (W_{}^{(l)} h^{(l - 1)} (x; θ)),

for all

l \in {1, \dots, H}

with

h^{(0)} (x; θ) = x

⁠. Here,

θ = vec ({[W^{(l)}]}_{l = 1}^{H + 1}) \in R^{d_{θ}}

with

W^{(l)} \in R^{d_{l} \times d_{l - 1}}

⁠,

d_{H + 1} = d_{y}

⁠, and

d_{0} = d_{x}

⁠. In addition,

σ^{(l)} : R^{d_{l}} \to R^{d_{l}}

represents an arbitrary nonlinear activation function per layer

l

and is allowed to differ among different layers.

4.2.2 Background

Given the difficulty of theoretically understanding deep neural networks, Goodfellow, Bengio, and Courville (2016) noted that theoretically studying simplified networks (i.e., deep linear networks) is worthwhile. For example, Saxe, McClelland, and Ganguli (2014) empirically showed that deep linear networks may exhibit several properties analogous to those of deep nonlinear networks. Accordingly, the theoretical study of deep linear neural networks has become an active area of research (Kawaguchi, 2016; Hardt & Ma, 2017; Arora, Cohen, Golowich, & Hu, 2018; Arora, Cohen, & Hazan, 2018; Bartlett, Helmbold, & Long, 2019; Du & Hu, 2019).

Along this line, Laurent and Brecht (2018) recently proved the following main result, proposition 2, under the assumptions PA4, PA5, and PA6:

PA4: Every activation function is identity as $σ^{(l)} (q) = q$ for every $l \in {1, \dots, H}$ (i.e., deep linear networks).
PA5: For any $y$ ⁠, the function $ℓ_{y}$ is convex and differentiable.
PA6: The thinnest layer is either the input layer or the output layer as $min {d_{x}, d_{y}} \leq min {d_{1}, \dots, d_{H}}$ ⁠.

Proposition 2

(Laurent & Brecht, 2018). Let $f$ be specified by equation 4.3. Let assumptions PA4, PA5, and PA6 hold. Then every local minimum $θ$ of $L$ is a global minimum.

4.2.3 Result

Instead of studying deep linear networks, we now consider a partial linear structure locally induced by a parameter vector with nonlinear activation functions. This relaxes the linearity assumption and extends our understanding of deep linear networks to deep nonlinear networks.

Intuitively, $J_{n, t} [θ]$ is a set of partial linear structures locally induced by a vector $θ$ ⁠, which is now formally defined as follows. Given a $θ \in R^{d_{θ}}$ ⁠, let $J_{n, t} [θ]$ be a set of all sets $J = {J^{(t + 1)}, \dots, J^{(H + 1)}}$ such that each set $J = {J^{(t + 1)}, \dots, J^{(H + 1)}} \in J_{n, t} [θ]$ satisfies the following conditions: there exists $ε > 0$ such that for all $l \in {t + 1, t + 2, \dots, H + 1}$ ⁠,

$J^{(l)} \subseteq {1, \dots, d_{l}}$ with $| J^{(l)} | \geq n$ ⁠.
$h^{(l)} {(x_{i}, θ^{'})}_{k} = {(W_{}^{(l)} h^{(l - 1)} (x_{i}, θ^{'}))}_{k}$ for all $(k, θ^{'}, i) \in J^{(l)} \times B (θ, ε) \times {1, \dots, m}$ ⁠.
$W_{i, j}^{(l + 1)} = 0$ for all $(i, j) \in ({1, \dots, d_{l + 1}} ∖ J^{(l + 1)}) \times J^{(l)}$ if $l \leq H - 1$ ⁠.

Let $Θ_{n, t}$ be the set of all parameter vectors $θ$ such that $J_{n, t} [θ]$ is nonempty. As the definition reveals, a neural network with a $θ \in Θ_{d_{y}, t}$ can be a standard deep nonlinear neural network (with no linear units).

Theorem 4.

Let

f

be specified by equation . Let assumption 1 hold. Then for any

t \in {1, \dots, H}

⁠, at every local minimum

θ \in (Θ_{d_{y}, t} ∖ \tilde{Ω})

of

L

⁠, the following holds. There exists

ε_{0} > 0

such that for any

ε \in (0, ε_{0})

⁠,

L (θ) = inf_{\begin{matrix} S \subseteq_{f i n} V [θ, ε], \\ α \in R^{d_{θ} \times | S |}, α_{h} \in R^{d_{t}} \end{matrix}} {\tilde{L}}_{θ, t}^{(f f)} (α, α_{h}, ε, S),

where

\begin{matrix} {\tilde{L}}_{θ, t}^{(f f)} (α, α_{h}, ε, S) = \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) + {\tilde{f}}_{θ} (x_{i}; α, ε, S)), \end{matrix}

with

α_{h} = v e c ({[α_{h}^{(l + 1)}]}_{l = t}^{H}) \in R^{d_{t}}

⁠,

α_{h}^{(l + 1)} \in R^{d_{y} \times d_{l}}

and

d_{t} = d_{y} \sum_{l = t}^{H} d_{l}

⁠.

Theorem 4 is a special case of theorem 2. A special case of theorem 4 then results in one of the main results in the literature regarding deep linear neural networks, that is, every local minimum is a global minimum. Consider any deep linear network with $d_{y} \leq min {d_{1}, \dots, d_{H}}$ ⁠. Then every local minimum $θ$ is in $Θ_{d_{y}, 0} ∖ \tilde{Ω} = Θ_{d_{y}, 0}$ ⁠. Hence, theorem 4 is reduced to the statement that for any local minimum, $L (θ) = {inf}_{α_{h} \in R^{d_{t}}} \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (\sum_{l = 0}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u)) = {inf}_{α_{x} \in R^{d_{x}}} \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (α_{x} x_{i})$ ⁠, which is the global minimum value. Thus, every local minimum is a global minimum for any deep linear neural network with $d_{y} \leq min {d_{1}, \dots, d_{H}}$ ⁠. Therefore, theorem 4 successfully generalizes the recent previous result in the literature (proposition 2) for a common scenario of $d_{y} \leq d_{x}$ ⁠.

Beyond deep linear networks, theorem 4 illustrates both the benefit of the locally induced structure and the overparameterization for deep nonlinear networks. In the first term, $\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u)$ ⁠, in $L_{θ, t}^{(ff)}$ ⁠, we benefit by decreasing $t$ (a more locally induced structure) and increasing the width of the $l$ th layer for any $l \geq t$ (overparameterization). The second term, ${\tilde{f}}_{θ} (x_{i}; α, ε, S)$ in $L_{θ, t}^{(ff)}$ ⁠, is the general term that is always present from theorem 2, where we benefit from increasing $d_{θ}$ because $α \in R^{d_{θ} \times | S |}$ ⁠.

From the geometric view, theorem 4 captures the intuition that the span ${\tilde{T}}_{f_{X} (θ)}$ of the set of the vectors in the tangent spaces ${T_{f_{X} (θ + ε v)} : v \in V [θ, ε]}$ contains the best basis model with the linear feature for deep linear networks, as well as the best basis models with more nonlinear features as more local structures arise. Similar to the examples in Figures 3 and 4, ${\tilde{T}}_{f_{X} (θ)} \neq T_{f (θ)}$ and the output of the best basis models with those features are contained in ${\tilde{T}}_{f_{X} (θ)}$ but not in $T_{f (θ)}$ ⁠.

A similar local structure was recently considered in Kawaguchi, Huang, and Kaelbling (2019). However, both the problem settings and the obtained results largely differ from those in Kawaguchi et al. (2019). Furthermore, theorem 4 is proven as a consequence of our general theory (theorem 2), and accordingly, the proofs largely differ from each other as well. Theorem 4 also differs from recent results on the gradient decent algorithm for deep linear networks (Arora, Cohen, Golowich, & Hu, 2018; Arora, Cohen, & Hazan, 2018; Bartlett et al., 2019; Du & Hu, 2019), since we analyze the loss surface instead of a specific algorithm and theorem 4 applies to deep nonlinear networks as well.

4.3 Proof Idea in Applications of Theorem 2

Theorems 3 and 4 are simple consequences of theorem 2, and their proof is illustrative as a means of using theorem 2 in future studies with different additional assumptions. The high-level idea behind the proofs in the applications of theorem 2 is captured in the geometric view of theorem 2 (see Figures 3 and 4). That is, given a desired guarantee, we check whether the space ${\tilde{T}}_{f_{X} (θ)}$ is expressive enough to contain the output of the desired model corresponding to the desired guarantee.

To simplify the use of theorem 2, we provide the following lemma. This lemma states that the expressivity of the model ${\tilde{f}}_{θ} (x; α, ε, S)$ with respect to $(α, S)$ is the same as that of ${\tilde{f}}_{θ} (x; α, ε, S) + {\tilde{f}}_{θ} (x; α^{'}, ε, S^{'})$ with respect to $(α, α^{'}, S, S^{'})$ ⁠. As shown in its proof, this is essentially because ${\tilde{f}}_{θ}$ is linear in $α$ ⁠, and a union of two sets $S \subseteq_{fin} V [θ, ε]$ and $S^{'} \subseteq_{fin} V [θ, ε]$ remains a finite subset of $V [θ, ε]$ ⁠.

Lemma 1.

For any $θ$ ⁠, any $ε \geq 0$ ⁠, any $S^{'} \subseteq_{f i n} V [θ, ε]$ ⁠, and any $x$ ⁠, it holds that ${{\tilde{f}}_{θ} (x; α, ε, S) : α \in R^{d_{θ} \times | S |}, S \subseteq_{f i n} V [θ, ε]} = {{\tilde{f}}_{θ} (x; α, ε, S) + {\tilde{f}}_{θ} (x; α^{'}, ε, S^{'}) : α \in R^{d_{θ} \times | S |}, α^{'} \in R^{d_{θ} \times | S^{'} |}, S \subseteq_{f i n} V [θ, ε]} .$

Based on theorem 2 and lemma 1, the proofs of theorems 3 and 4 are reduced to a simple search for finding $S^{'} \subseteq_{fin} V [θ, ε]$ such that the expressivity of ${\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'})$ with respect to $α^{'}$ is no worse than the expressivity of $α_{w} x_{i} + α_{r} z (x_{i}; u)$ with respect to $(α_{w}, α_{r})$ (see theorem 3) and that of $\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u)$ with respect to $α_{h}^{(l + 1)}$ (see theorem 4). In other words, ${{\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}} \supseteq {α_{w} x_{i} + α_{r} z (x_{i}; u) : α_{w} \in R^{d_{y} \times d_{x}}, α_{r} \in R^{d_{y} \times d_{z}}}$ (see theorem 3) and ${{\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}} \supseteq {\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) : α_{h} \in R^{d_{t}}}$ (see theorem 4). Only with this search for $S^{'}$ ⁠, theorem 2 together with lemma 1 implies the desired statements for theorems 3 and 4 (see sections A.4 and A.5 in the appendix for further details). Thus, theorem 2 also enables simple proofs.

5 Conclusion

This study provided a general theory for nonconvex machine learning and demonstrated its power by proving new competitive theoretical results with it. In general, the proposed theory provides a mathematical tool to study the effects of hypothesis classes $f$ ⁠, methods, and assumptions through the lens of the global optima of the perturbable gradient basis model class.

In convex machine learning with a model output $f (x; θ) = θ^{⊤} x$ with a (nonlinear) feature output $x = φ (x^{(raw)})$ ⁠, achieving a critical point ensures the global optimality in the span of the fixed basis $x = φ (x^{(raw)})$ ⁠. In nonconvex machine learning, we have shown that achieving a critical point ensures the global optimality in the span of the gradient basis $\partial f_{x} (θ)$ ⁠, which coincides with the fixed basis $x = φ (x^{(raw)})$ in the case of the convex machine learning. Thus, whether convex or nonconvex, achieving a critical point ensures the global optimality in the span of some basis, which might be arbitrarily bad (or good) depending on the choice of the handcrafted basis $φ (x^{(raw)}) = \partial f_{x} (θ)$ (for the convex case) or the induced basis $\partial f_{x} (θ)$ (for the nonconvex case). Therefore, in terms of the loss values at critical points, nonconvex machine learning is theoretically as justified as the convex one, except in the case when a preference is given to $φ (x^{(raw)})$ over $\partial f_{x} (θ)$ (both of which can be arbitrarily bad or good). The same statement holds for local minima and perturbable gradient basis.

Appendix: Proofs of Theoretical Results

In this appendix, we provide complete proofs of the theoretical results.

A.1 Proof of Theorem 1

The proof of theorem 1 combines lemma 2 with assumptions 1 and 2 by taking advantage of the structure of the objective function $L$ ⁠. Although lemma 2 is rather weak and assumptions 1 and 2 are mild (in the sense that they usually hold in practice), a right combination of these with the structure of $L$ can prove the desired statement.

Lemma 2.

Assume that for any

i \in {1, \dots, m}

⁠, the function

ℓ_{y_{i}} : q \mapsto ℓ (q, y_{i})

is differentiable. Then for any critical point

θ \in (R^{d_{θ}} ∖ Ω)

of

L

⁠, the following holds: for any

k \in {1, \dots, d_{θ}},

\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ) = 0 .

Proof of Lemma ².

Let $θ$ be an arbitrary critical point $θ \in (R^{d_{θ}} ∖ Ω)$ of $L$ ⁠. Since $ℓ_{y_{i}} : R^{d_{y}} \to R$ is assumed to be differentiable and $f_{x_{i}} \in R^{d_{y}}$ is differentiable at the given $θ$ ⁠, the composition $(ℓ_{y_{i}} \circ f_{x_{i}})$ is also differentiable, and $\partial_{k} (ℓ_{y_{i}} \circ f_{x_{i}}) = \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ)$ ⁠. In addition, $L$ is differentiable because a sum of differentiable functions is differentiable. Therefore, for any critical point $θ$ of $L$ ⁠, we have that $\partial L (θ) = 0$ ⁠, and, hence, $\partial_{k} L (θ) = \sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ) = 0,$ for any $k \in {1, \dots, d_{θ}},$ from linearity of differentiation operation. $□$

Proof of Theorem ¹.

Let

θ \in (R^{d_{θ}} ∖ Ω)

be an arbitrary critical point of

L

⁠. From assumption 2, there exists a function

g

such that

f_{x_{i}} (θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ)

for all

i \in {1, \dots, m}

⁠. Then, for any

α \in R^{d_{θ}}

⁠,

\begin{matrix} L_{θ} (α) & \geq & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) + λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) (f_{θ} (x_{i}; α) - f (x_{i}; θ)) \\ = & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) + \sum_{k = 1}^{d_{θ}} α_{k} \underset{= 0 from Lemma 2}{\underset{︸}{\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ)}} \\ - \sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) f (x_{i}; θ) \\ = & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) - \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \underset{= 0 from Lemma 2}{\underset{︸}{\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ)}}, \\ = & L (θ), \end{matrix}

where the first line follows from assumption 1 (differentiable and convex

ℓ_{y_{i}}

⁠), the second line follows from linearity of summation, and the third line follows from assumption 2. Thus, on the one hand, we have that

L (θ) \leq {inf}_{α \in R^{d_{θ}}} L_{θ} (α)

⁠. On the other hand, since

f (x_{i}; θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ) \in {f_{θ} (x_{i}; α) = \sum_{k = 1}^{d_{θ}} α_{k} \partial_{k} f_{x_{i}} (θ) : α \in R^{d_{θ}}}

⁠, we have that

L (θ) \geq {inf}_{α \in R^{d_{θ}}} L_{θ} (α)

⁠. Combining these yields the desired statement of

L (θ) = {inf}_{α \in R^{d_{θ}}} L_{θ} (α)

⁠.

□

A.2 Proof of Theorem 2

The proof of theorem 2 uses lemma 3, the structure of the objective function $L$ ⁠, and assumptions 1 and 2.

Lemma 3.

Assume that for any

i \in {1, \dots, m}

⁠, the function

ℓ_{y_{i}} : q \mapsto ℓ (q, y_{i})

is differentiable. Then for any local minimum

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

of

L

⁠, the following holds: there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠, any

v \in V [θ, ε]

⁠, and any

k \in {1, \dots, d_{θ}}

⁠,

\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ + ε v) = 0 .

Proof of Lemma ³.

Let

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

be an arbitrary local minimum of

L

⁠. Since

θ

is a local minimum of

L

⁠, by the definition of a local minimum, there exists

ε_{1} > 0

such that

L (θ) \leq L (θ^{'})

for all

θ^{'} \in B (θ, ε_{1})

⁠. Then for any

ε \in [0, ε_{1} / 2)

and any

ν \in V [θ, ε]

⁠, the vector

(θ + ε v)

is also a local minimum because

L (θ + ε v) = L (θ) \leq L (θ^{'})

for all

θ^{'} \in B (θ + ε v, ε_{1} / 2) \subseteq B (θ, ε_{1})

(the inclusion follows from the triangle inequality), which satisfies the definition of a local minimum for

(θ + ε v)

⁠.

Since $θ \in (R^{d_{θ}} ∖ \tilde{Ω})$ ⁠, there exists $ε_{2} > 0$ such that $f_{x_{1}}, \dots, f_{x_{m}}$ are differentiable in $B (θ, ε_{2})$ ⁠. Since $ℓ_{y_{i}} : R^{d_{y}} \to R$ is assumed to be differentiable and $f_{x_{i}} \in R^{d_{y}}$ is differentiable in $B (θ, ε_{2})$ ⁠, the composition $(ℓ_{y_{i}} \circ f_{x_{i}})$ is also differentiable, and $\partial_{k} (ℓ_{y_{i}} \circ f_{x_{i}}) = \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ)$ in $B (θ, ε_{2})$ ⁠. In addition, $L$ is differentiable in $B (θ, ε_{2})$ because a sum of differentiable functions is differentiable.

Therefore, with

ε_{0} = min (ε_{1} / 2, ε_{2})

⁠, we have that for any

ε \in [0, ε_{0})

and any

ν \in V [θ, ε]

⁠, the vector

(θ + ε v)

is a differentiable local minimum, and hence the first-order necessary condition of differentiable local minima implies that

\partial_{k} L (θ + ε v) = \sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ + ε v) = 0,

for any

k \in {1, \dots, d_{θ}},

where we used the fact that

f_{x_{i}} (θ) = f_{x_{i}} (θ + ε v)

for any

v \in V [θ, ε]

⁠.

□

Proof of Theorem ².

Let

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

be an arbitrary local minimum of

L

⁠. Since

(R^{d_{θ}} ∖ \tilde{Ω}) \subseteq (R^{d_{θ}} ∖ Ω)

⁠, from assumption 2, there exists a function

g

such that

f_{x_{i}} (θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ)

for all

i \in {1, \dots, m}

⁠. Then from lemma 3, there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠, any

S \subseteq_{fin} V [θ, ε]

and any

α \in R^{d_{θ} \times | S |}

⁠,

\begin{matrix} {\tilde{L}}_{θ} (α, ε, S) & \geq & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) + λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) ({\tilde{f}}_{θ} (x_{i}; α, ε, S) - f (x_{i}; θ)) \\ = & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) + \sum_{k = 1}^{d_{θ}} \sum_{j = 1}^{| S |} α_{k, j} \underset{= 0 from Lemma 3}{\underset{︸}{\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ + ε S_{j})}} \\ - \sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) f (x_{i}; θ) \\ = & \sum_{i = 1}^{m} λ_{i} ℓ_{y_{i}} (f_{x_{i}} (θ)) - \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \underset{= 0 from Lemma 3}{\underset{︸}{\sum_{i = 1}^{m} λ_{i} \partial ℓ_{y_{i}} (f_{x_{i}} (θ)) \partial_{k} f_{x_{i}} (θ)}}, \\ = & L (θ), \end{matrix}

where the first line follows from assumption 1 (differentiable and convex

ℓ_{y_{i}}

⁠), the second line follows from linearity of summation and the definition of

{\tilde{f}}_{θ} (x_{i}; α, ε, S)

⁠, and the third line follows from assumption 2. Thus, on the one hand, there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠,

L (θ) \leq inf {{\tilde{L}}_{θ} (α, ε, S) : S \subseteq_{fin} V [θ, ε], α \in R^{d_{θ} \times | S |}}

⁠. On the other hand, since

f (x_{i}; θ) = \sum_{k = 1}^{d_{θ}} g {(θ)}_{k} \partial_{k} f_{x_{i}} (θ) \in {{\tilde{f}}_{θ} (x_{i}; α, ε, S) : α \in R^{d_{θ}}, S = 0}

⁠, we have that

L (θ) \leq inf {{\tilde{L}}_{θ} (α, ε, S) : S \subseteq_{fin} V [θ, ε], α \in R^{d_{θ} \times | S |}}

⁠. Combining these yields the desired statement.

□

A.3 Proof of Lemma 1

As shown in the proof of lemma 1, lemma 1 is a simple consequence of the following facts: ${\tilde{f}}_{θ}$ is linear in $α$ and a union of two sets $S \subseteq_{fin} V [θ, ε]$ and $S^{'} \subseteq_{fin} V [θ, ε]$ is still a finite subset of $V [θ, ε]$ ⁠.

Proof of Lemma ¹.

Let

S^{'} \subseteq_{fin} V [θ, ε]

be fixed. Then,

\begin{matrix} {{\tilde{f}}_{θ} (x; α, ε, S) : α \in R^{d_{θ} \times | S |}, S \subseteq_{fin} V [θ, ε]} \\ = {{\tilde{f}}_{θ} (x; α, ε, S \cup S^{'}) : α \in R^{d_{θ} \times | S \cup S^{'} |}, S \subseteq_{fin} V [θ, ε]} \\ = {{\tilde{f}}_{θ} (x; α, ε, S ∖ S^{'}) + {\tilde{f}}_{θ} (x; α^{'}, ε, S^{'}) : α \in R^{d_{θ} \times | S ∖ S^{'} |}, α^{'} \in R^{d_{θ} \times | S^{'} |}, \\ S \subseteq_{fin} V [θ, ε]} \\ = {{\tilde{f}}_{θ} (x; α, ε, S \cup S^{'}) + f_{θ} (x; α^{'}, ε, S^{'}) : α \in R^{d_{θ} \times | S \cup S^{'} |}, α^{'} \in R^{d_{θ} \times | S^{'} |}, \\ S \subseteq_{fin} V [θ, ε]} \\ = {{\tilde{f}}_{θ} (x; α, ε, S) + f_{θ} (x; α^{'}, ε, S^{'}) : α \in R^{d_{θ} \times | S |}, α^{'} \in R^{d_{θ} \times | S^{'} |}, \\ S \subseteq_{fin} V [θ, ε]}, \end{matrix}

where the second line follows from the facts that a finite union of finite sets is finite and hence

S \cup S^{'} \subseteq_{fin} V [θ, ε]

(i.e., the set in the first line is a superset of

\supseteq,

the set in the second line), and that

α \in R^{d_{θ} \times | S \cup S^{'} |}

can vanish the extra terms due to

S^{'}

in

{\tilde{f}}_{θ} (x; α, ε, S \cup S^{'})

(i.e., the set in the first line is a subset of,

\subseteq,

the set in the second line). The last line follows from the same facts. The third line follows from the definition of

{\tilde{f}}_{θ} (x; α, ε, S)

⁠. The fourth line follows from the following equality due to the linearity of

{\tilde{f}}_{θ}

in

α

⁠:

\begin{matrix} {{\tilde{f}}_{θ} (x; α^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}} \\ = \{\sum_{k = 1}^{d_{θ}} \sum_{j = 1}^{| S |} (α_{k, j}^{'} + {\bar{α}}_{k, j}^{'}) \partial_{k} f_{x} (θ + ε S_{j}^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}, {\bar{α}}^{'} \in R^{d_{θ} \times | S^{'} |}\} \\ = {{\tilde{f}}_{θ} (x; α^{'}, ε, S^{'}) + {\tilde{f}}_{θ} (x; {\bar{α}}^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}, {\bar{α}}^{'} \in R^{d_{θ} \times | S^{'} |}} . \end{matrix}

□

A.4 Proof of Theorem 3

As shown in the proof of theorem 3, thanks to theorem 2 and lemma 1, the remaining task to prove theorem 3 is to find a set $S^{'} \subseteq_{fin} V [θ, ε]$ such that ${{\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}} \supseteq {α_{w} x_{i} + α_{r} z (x_{i}; u) : α_{w} \in R^{d_{y} \times d_{x}}, α_{r} \in R^{d_{y} \times d_{z}}}$ ⁠. Let $Null (M)$ be the null space of a matrix $M$ ⁠.

Proof of Theorem ³.

Let

θ \in (R^{d_{θ}} ∖ \tilde{Ω})

be an arbitrary local minimum of

L

⁠. Since

f

is specified by equation , and hence

f (x; θ) = (\partial_{vec (W)} f (x; θ)) vec (W)

⁠, assumption 2 is satisfied. Thus, from theorem 2, there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠,

L (θ) = inf_{\begin{matrix} S \subseteq_{fin} V [θ, ε], α \in R^{d_{θ} \times | S |} \end{matrix}} \sum_{i = 1}^{m} λ_{i} ℓ ({\tilde{f}}_{θ} (x_{i}; α, ε, S), y_{i}),

where

\begin{matrix} {\tilde{f}}_{θ} (x_{i}; α, ε, S) & = & \sum_{j = 1}^{| S |} α_{w, j} (x_{i} + (R + ε v_{r, j}) z_{i, j}) + (W + ε v_{w, j}) α_{r, j} z_{i, j} \\ + (\partial_{u} f_{x_{i}} (θ + ε S_{j})) α_{u, j}, \end{matrix}

with

α = [α_{\cdot 1}, \dots, α_{\cdot | S |}] \in R^{d_{θ} \times | S |}

⁠,

α_{\cdot j} = vec ([α_{w, j}, α_{r, j}, α_{u, j}]) \in R^{d_{θ}}

⁠,

S_{j} = vec ([v_{w, j}, v_{r, j}, v_{u, j}]) \in R^{d_{θ}}

⁠, and

z_{i, j} = z (x_{i}, u + ε v_{u, j})

for all

j \in {1, \dots, | S |}

⁠. Here,

α_{w, j}, v_{w, j} \in R^{d_{y} \times d_{x}}

⁠,

α_{r, j}, v_{r, j} \in R^{d_{x} \times d_{z}}

⁠, and

α_{u, j}, v_{u, j} \in R^{d_{u}}

⁠. Let

ε \in (0, ε_{0})

be fixed.

Consider the case of

rank (W) \geq d_{y}

⁠. Define

\bar{S}

such that

| \bar{S} | = 1

and

{\bar{S}}_{1} = 0 \in R^{d_{θ}}

⁠, which is in

V [θ, ε]

⁠. Then by setting

α_{u, 1} = 0

and rewriting

α_{r, 1}

such that

W α_{r, 1} = α_{r, 1}^{(1)} - α_{w, 1} R

with an arbitrary matrix

α_{r, 1} \in R^{d_{y} \times d_{z}}

(this is possible since

rank (W) \geq d_{y}

⁠), we have that

\begin{matrix} {{\tilde{f}}_{θ} (x_{i}; α, ε, \bar{S}) : α \in R^{d_{θ} \times | \bar{S} |}} \\ \supseteq {α_{w, 1} x_{i} + α_{r, 1}^{(1)} z_{i, 1} : α_{w, 1} \in R^{d_{y} \times d_{x}}, α_{r, 1}^{(1)} \in R^{d_{y} \times d_{z}}} . \end{matrix}

Consider the case of

rank (W) < d_{y}

⁠. Since

W \in R^{d_{y} \times d_{x}}

and

rank (W) < d_{y} \leq min (d_{x}, d_{z}) \leq d_{x}

⁠, we have that

Null (W) \neq {0}

⁠, and there exists a vector

a \in R^{d_{x}}

such that

a_{} \in Null (W)

and

{∥ a ∥}_{2} = 1

⁠. Let

a

be such a vector. Define

{\bar{S}}^{'}

as follows:

| {\bar{S}}^{'} | = d_{y} d_{z} + 1

⁠,

{\bar{S}}_{1}^{'} = 0 \in R^{d_{θ}}

⁠, and set

{\bar{S}}_{j}^{'}

for all

j \in {2, \dots, d_{y} d_{z} + 1}

such that

v_{w, j} = 0

⁠,

v_{u, j} = 0

⁠, and

v_{r, j} = a b_{j}^{⊤}

where

b_{j} \in R^{d_{z}}

is an arbitrary column vector with

∥ b_{j} ∥_{2} \leq 1

⁠. Then

{\bar{S}}_{j}^{'} \in V [θ, ε]

for all

j \in {1, \dots, d_{y} d_{z} + 1}

⁠. By setting

α_{r, j} = 0

and

α_{u, j} = 0

for all

j \in {1, \dots, d_{y} d_{z} + 1}

and by rewriting

α_{w, 1} = α_{w, 1}^{(1)} - \sum_{j = 2}^{d_{y} d_{z} + 1} α_{w, j}

and

α_{w, j} = \frac{1}{ε} q_{j} a^{T}

for all

j \in {2, \dots, d_{y} d_{z} + 1}

with an arbitrary vector

q_{j} \in R^{d_{y}}

(this is possible since

ε > 0

is fixed first and

α_{w, j}

is arbitrary), we have that

\begin{matrix} {{\tilde{f}}_{θ} (x_{i}; α, ε, {\bar{S}}^{'}) : α \in R^{d_{θ} \times | {\bar{S}}^{'} |}} \\ \supseteq \{α_{w, 1}^{(1)} x_{i} + (α_{w, 1}^{(1)} R + \sum_{j = 2}^{d_{y} d_{z} + 1} q_{j} b_{j}^{⊤}) z_{i, 1} : q_{j} \in R^{d_{y}}, b_{j} \in R^{d_{z}}\} . \end{matrix}

Since

q_{j} \in R^{d_{y}}

and

b_{j} \in R^{d_{z}}

are arbitrary, we can rewrite

\sum_{j = 2}^{d_{y} d_{z} + 1} q_{j} b_{j} = α_{w, 1}^{(2)} - α_{w, 1}^{(1)} R

with an arbitrary matrix

α_{w, 1}^{(2)} \in R^{d_{y} \times d_{z}}

⁠, yielding

\begin{matrix} {{\tilde{f}}_{θ} (x_{i}; α, ε, {\bar{S}}^{'}) : α \in R^{d_{θ} \times | {\bar{S}}^{'} |}} \\ \supseteq {α_{w, 1}^{(1)} x_{i} + α_{w, 1}^{(2)} z_{i, 1} : α_{w, 1}^{(1)} \in R^{d_{y} \times d_{x}}, α_{w, 1}^{(2)} \in R^{d_{y} \times d_{z}}} . \end{matrix}

By summarizing above, in both cases of

rank (W)

⁠, there exists a set

S^{'} \subseteq_{fin} V [θ, ε]

such that

\begin{matrix} {{\tilde{f}}_{θ} (x_{i}; α, ε, S) : α \in R^{d_{θ} \times | S |}, S \subseteq_{fin} V [θ, ε]} \\ = {{\tilde{f}}_{θ} (x_{i}; α, ε, S) + {\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'}) \\ : α \in R^{d_{θ} \times | S |}, α^{'} \in R^{d_{θ} \times | S^{'} |}, S \subseteq_{fin} V [θ, ε]} \\ \supseteq {{\tilde{f}}_{θ} (x_{i}; α, ε, S) + α_{w} x_{i} + α_{r} z (x_{i}, u) \\ : α \in R^{d_{θ} \times | S |}, α_{w}^{(1)} \in R^{d_{y} \times d_{x}}, α_{r}^{(2)} \in R^{d_{y} \times d_{z}}, S \subseteq_{fin} V [θ, ε]}, \end{matrix}

where the second line follows from lemma 1. On the other hand, since the set in the first line is a subset of the set in the last line,

{{\tilde{f}}_{θ} (x_{i}; α, ε, S) : α \in R^{d_{θ} \times | S |}, S \subseteq_{fin} V [θ, ε]} = {{\tilde{f}}_{θ} (x_{i}; α, ε, S) + α_{w} x_{i} + α_{r} z (x_{i}, u) : α \in R^{d_{θ} \times | S |}, α_{w}^{(1)} \in R^{d_{y} \times d_{x}}, α_{r}^{(2)} \in R^{d_{y} \times d_{z}}, S \subseteq_{fin} V [θ, ε]}

⁠. This immediately implies the desired statement from theorem 2.

□

A.5 Proof of Theorem 4

As shown in the proof of theorem 4, thanks to theorem 2 and lemma 1, the remaining task to prove theorem 4 is to find a set $S^{'} \subseteq_{fin} V [θ, ε]$ such that ${{\tilde{f}}_{θ} (x_{i}; α^{'}, ε, S^{'}) : α^{'} \in R^{d_{θ} \times | S^{'} |}} \supseteq {\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) : α_{h} \in R^{d_{t}}}$ ⁠. Let $M^{(l^{'})} \dots M^{(l + 1)} M^{(l)} = I$ if $l > l^{'}$ ⁠.

Proof of Theorem ⁴.

Since

f

is specified by equation and, hence,

f (x; θ) = (\partial_{vec (W^{(H + 1)})} f (x; θ)) vec (W^{(H + 1)}),

assumption 2 is satisfied. Let

t \in {0, \dots, H}

be fixed. Let

θ \in (Θ_{d_{y}, t} ∖ \tilde{Ω})

be an arbitrary local minimum of

L

⁠. Then from theorem 2, there exists

ε_{0} > 0

such that for any

ε \in [0, ε_{0})

⁠,

L (θ) = {inf}_{S \subseteq_{fin} V [θ, ε], α \in R^{d_{θ} \times | S |}} \sum_{i = 1}^{m} λ_{i} ℓ ({\tilde{f}}_{θ} (x_{i}; α, ε, S), y_{i}),

where

{\tilde{f}}_{θ} (x_{i}; α, ε, S) = \sum_{k = 1}^{d_{θ}} \sum_{j = 1}^{| S |} α_{k, j} \partial_{k} f_{x_{i}} (θ + ε S_{j}) .

Let

J = {J^{(t + 1)}, \dots, J^{(H + 1)}} \in J_{n, t} [θ]

be fixed. Without loss of generality, for simplicity of notation, we can permute the indices of the units of each layer such that

J^{(t + 1)}, \dots, J^{(H + 1)} \supseteq {1, \dots, d_{y}}

⁠. Let

\tilde{B} (θ, ε_{1}) = B (θ, ε_{1}) \cap {θ^{'} \in R^{d_{θ}} : W_{i, j}^{(l + 1)} = 0

for all

l \in {t + 1, \dots, H - 1}

and all

(i, j) \in ({1, \dots, d_{l + 1}} ∖ J^{(l + 1)}) \times J^{(l)}}

⁠. Because of the definition of the set

J

⁠, in

\tilde{B} (θ, ε_{1})

with

ε_{1} > 0

being sufficiently small, we have that for any

l \in {t, \dots, H}

⁠,

f_{x_{i}} (θ) = A^{(H + 1)} \dots A^{(l + 2)} [\begin{matrix} A^{(l + 1)} & C^{(l + 1)} \end{matrix}] h^{(l)} (x_{i}; θ) + ϕ_{x_{i}}^{(l)} (θ),

where

ϕ_{x_{i}}^{(l)} (θ) = \sum_{l^{'} = l}^{H - 1} A^{(H + 1)} \dots A^{(l^{'} + 3)} C^{(l^{'} + 2)} {\tilde{h}}^{(l^{'} + 1)} (x_{i}; θ)

and

{\tilde{h}}^{(l)} (x_{i}; θ) = σ^{(l)} (B^{(l)} {\tilde{h}}^{(l - 1)} (x_{i}; θ)),

for all

l \geq t + 2

with

{\tilde{h}}^{(t + 1)} (x_{i}; θ) = σ^{(t + 1)} ([\begin{matrix} ξ^{(l)} & B^{(l)} \end{matrix}] h^{(t)} (x_{i}; θ))

⁠. Here,

[\begin{matrix} A^{(l)} & C^{(l)} \\ ξ^{(l)} & B^{(l)} \end{matrix}] = W^{(l)}

with

A^{(l)} \in R^{d_{y} \times d_{y}}

⁠,

C^{(l)} \in R^{d_{y} \times (d_{l - 1} - d_{y})}

⁠,

B^{(l)} \in R^{(d_{l} - d_{y}) \times (d_{l - 1} - d_{y})}

⁠, and

ξ^{(l)} \in R^{(d_{l} - d_{y}) \times d_{y}}

⁠. Let

ε_{1} > 0

be a such number, and let

ε \in (0, min (ε_{0}, ε_{1} / 2))

be fixed so that both the equality from theorem 2 and the above form of

f_{x_{i}}

hold in

\tilde{B} (θ, ε)

⁠. Let

R^{(l)} = [\begin{matrix} A^{(l)} & C^{(l)} \end{matrix}]

⁠.

We will now find sets

S^{(t)}, \dots, S^{(H)} \subseteq_{fin} V [θ, ε]

such that

{{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) : α \in R^{d_{θ}}} \supseteq {α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) : α_{h}^{(l + 1)} \in R^{d_{y} \times d_{l}}} .

Find $S^{(l)}$ with $l = H$ ⁠: Since
$(\partial_{vec (R^{(H + 1)})} f_{x_{i}} (θ)) vec (α_{h}^{(H + 1)}) = α_{h}^{(H + 1)} h^{(H)} (x_{i}; θ),$
$S^{(H)} = {0} \subseteq_{fin} V [θ, ε]$ (where $0 \in R^{d_{θ}}$ ⁠) is the desired set.
Find $S^{(l)}$ with $l \in {t, \dots, H - 1}$ ⁠: With $α_{r}^{(l + 1)} \in R^{d_{l + 1} \times d_{l}}$ ⁠, we have that
$\begin{matrix} (\partial_{vec (R^{(l + 1)})} f_{x_{i}} (θ)) vec (α_{r}^{(l + 1)}) = A^{(H + 1)} \dots A^{(l + 2)} α_{r}^{(l + 1)} h^{(l)} (x_{i}; θ) . \end{matrix}$

Therefore, if

rank (A^{(H + 1)} \dots A^{(l + 2)}) \geq d_{y}

⁠, since

{A^{(H + 1)} \dots A^{(l + 2)} α_{r}^{(l + 1)} : α_{r}^{(l + 1)} \in R^{d_{l + 1} \times d_{l}}} \supseteq {α_{h}^{(l + 1)} \in R^{d_{y} \times d_{l}}}

⁠,

S^{(l)} = {0} \subseteq_{fin} V [θ, ε]

(where

0 \in R^{d_{θ}}

⁠) is the desired set. Let us consider the remaining case: let

rank (A^{(H + 1)} \dots A^{(l + 2)}) < d_{y}

and let

l \in {t, \dots, H - 1}

be fixed. Let

l^{*} = min {l^{'} \in Z^{+} : l + 3 \leq l^{'} \leq H + 2 \land rank (A^{(H + 1)} \dots A^{(l^{'})}) \geq d_{y}},

where

A^{(H + 1)} \dots A^{(H + 2)} ≜ I_{d_{y}}

and the minimum exists since the set is finite and contains at least

H + 2

(nonempty). Then

rank (A^{(H + 1)} \dots A^{(l^{*})}) \geq d_{H + 1}

and

rank (A^{(H + 1)} \dots A^{(l^{'})}) < d_{H + 1}

for all

l^{'} \in {l + 2, l + 3, \dots, l^{*} - 1}

⁠. Thus, for all

l^{'} \in {l + 1, l + 2, \dots, l^{*} - 2}

⁠, there exists a vector

a_{l^{'}} \in R^{d_{y}}

such that

a_{l^{'}} \in Null (A^{(H + 1)} \dots A^{(l^{'} + 1)}) and {∥ a_{l^{'}} ∥}_{2} = 1 .

Let

a_{l^{'}}

denote such a vector. Consider

S^{(l)}

such that the weight matrices

W

are perturbed with

\bar{θ} + ε S_{j}^{(l)}

as

{\tilde{A}}_{j}^{(l^{'})} = A^{(l^{'})} + ε a_{l^{'}} b_{l^{'}, j}^{⊤} and {\tilde{R}}_{j}^{(l + 1)} = R^{(l + 1)} + ε a_{l + 1} b_{l + 1, j}^{⊤}

for all

l^{'} \in {l + 2, l + 3, \dots, l^{*} - 2}

⁠, where

∥ b_{l^{'}, j} ∥_{2}

is bounded such that

∥ S_{j}^{(l)} ∥_{2} \leq 1

⁠. That is, the entries of

S_{j}

are all zeros except the entries corresponding to

A^{(l^{'})}

(for

l^{'} \in {l + 2, l + 3, \dots, l^{*} - 2}

⁠) and

R^{(l + 1)}

⁠. Then

S_{j}^{(l)} \in V [θ, ε]

⁠, since

A^{(H + 1)} \dots A^{(l^{'} + 1)} {\tilde{A}}_{j}^{(l^{'})} = A^{(H + 1)} \dots A^{(l^{'} + 1)} A^{(l^{'})}

for all

l^{'} \in {l + 2, l + 3, \dots, l^{*} - 2}

and

A^{(H + 1)} \dots A^{(l + 2)} {\tilde{R}}_{j}^{(l + 1)} = A^{(H + 1)} \dots A^{(l + 2)} R^{(l + 1)}

⁠. Let

| S^{(l)} | = 2 N

with some integer

N

to be chosen later. Define

S_{j + N}^{(l)}

for

j = 1, \dots, N

by setting

S_{j + N}^{(l)} = S_{j}^{(l)}

except that

b_{l + 1, j + N} = 0

whereas

b_{l + 1, j}

is not necessarily zero. By setting

α_{j + N} = - α_{j}

for all

j \in {1, \dots, N}

⁠, with

α_{j} \in R^{d_{l^{*}} \times d_{l^{*} - 1}}

⁠,

\begin{matrix} {\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) \\ = \sum_{j = 1}^{N} A^{(H + 1)} \dots A^{(l^{*})} (α_{j} + α_{j + N}) {\tilde{A}}^{(l^{*} - 2)} \dots {\tilde{A}}^{(l + 2)} R^{(l + 1)} h^{(l)} (x_{i}; θ) \\ + \sum_{j = 1}^{N} (\partial_{vec (A^{(l^{*} - 1)})} ϕ_{x_{i}}^{(l)} (θ + ε S_{j})) vec (α_{j} + α_{j + N}) \\ + ε \sum_{j = 1}^{N} A^{(H + 1)} \dots A^{(l^{*})} α_{j} {\tilde{A}}^{(l^{*} - 2)} \dots {\tilde{A}}^{(l + 2)} a_{l + 1} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ) \\ = ε \sum_{j = 1}^{N} A^{(H + 1)} \dots A^{(l^{*})} α_{j} {\tilde{A}}^{(l^{*} - 2)} \dots {\tilde{A}}^{(l + 2)} a_{l + 1} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ), \end{matrix}

where we used the fact that

\partial_{vec (A^{(l^{*} - 1)})} ϕ_{x_{i}}^{(l)} (θ + ε S_{j})

does not contain

b_{l + 1, j}

⁠. Since

rank (A^{(H + 1)} \dots A^{(l^{*})}) \geq d_{y}

and

{A^{(H + 1)} \dots A^{(l^{*})} α_{j} : α_{j} \in R^{d_{l^{*}} \times d_{l^{*} - 1}}} = {\frac{1}{ε} α_{j}^{'} : α_{j}^{'} \in R^{d_{y} \times d_{l^{*} - 1}}}

⁠, we have that

\forall α_{j}^{'} \in R^{d_{y} \times d_{l^{*} - 1}}

⁠,

\exists α \in R^{d_{θ} \times | S |}

⁠,

{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) = \sum_{j = 1}^{N} α_{j}^{'} {\tilde{A}}^{(l^{*} - 2)} \dots {\tilde{A}}^{(l + 2)} a_{l + 1} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ) .

Let

N = 2 N_{1}

⁠. Define

S_{j + N_{1}}^{(l)}

for

j = 1, \dots, N_{1}

by setting

S_{j + N_{1}}^{(l)} = S_{j}^{(l)}

except that

b_{l^{*} - 2, j + N_{1}} = 0

⁠, whereas

b_{l^{*} - 2, j}

is not necessarily zero. By setting

α_{j + N_{1}}^{'} = - α_{j}^{'}

for all

j \in {1, \dots, N_{1}}

⁠,

{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) = ε \sum_{j = 1}^{N_{1}} α_{j}^{'} a_{l^{*} - 2} b_{l^{*} - 2, j}^{⊤} {\tilde{A}}^{(l^{*} - 3)} \dots {\tilde{A}}^{(l + 2)} a_{l + 1} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ) .

By induction,

{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) = ε^{t} \sum_{j = 1}^{N_{t}} α_{j}^{'} a_{l^{*} - 2} b_{l^{*} - 2, j} a_{l^{*} - 3} b_{l^{*} - 3, j} \dots a_{l + 1} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ),

where

t = (l^{*} - 2) - (l + 2) + 1

is finite. By setting

α_{j}^{'} = \frac{1}{ε^{t}} q_{j} a_{l^{*} - 2}^{⊤}

and

b_{l, j} = a_{l - 1}

for all

l = l^{*} - 2, \dots, l

(⁠

ε > 0

⁠),

{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) = \sum_{j = 1}^{N_{t}} q_{j} b_{l + 1, j}^{⊤} h^{(l)} (x_{i}; θ) .

Since

q_{j} b_{l + 1, j}

are arbitrary, with sufficiently large

N_{t}

(⁠

N_{t} = d_{y} d_{l}

suffices), we can set

\sum_{j = 1}^{N_{t}} q_{j} b_{l + 1, j} = α_{h}^{(l)}

for any

α_{h}^{(l)} \in R^{d_{θ} \times d_{l}}

⁠, and hence

{{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) : α \in R^{d_{θ} \times | S^{(l)} |}} \supseteq {α_{h}^{(l)} h^{(l)} (x_{i}; θ) : α_{h}^{(l)} \in R^{d_{θ} \times d_{l}}} .

Thus far, we have found the sets

S^{(t)}, \dots, S^{(H)} \subseteq_{fin} V [θ, ε]

such that

{{\tilde{f}}_{θ} (x_{i}; α, ε, S^{(l)}) : α \in R^{d_{θ}}} \supseteq {α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) : α_{h}^{(l + 1)} \in R^{d_{y} \times d_{l}}}

⁠. From lemma 1, we can combine these, yielding

\begin{matrix} {{\tilde{f}}_{θ} (x_{i}; α, ε, S^{}) : α \in R^{d_{θ}}, S \subseteq_{fin} V [θ, ε]} \\ = \{\sum_{l = t}^{H} {\tilde{f}}_{θ} (x_{i}; α^{(l)}, ε, S^{(l)}) + {\tilde{f}}_{θ} (x_{i}; α, ε, S^{}) : α^{(t)}, \dots, α^{(H)} \in R^{d_{θ}}, \\ α \in R^{d_{θ}}, S \subseteq_{fin} V [θ, ε]\} \\ \supseteq \{\sum_{l = t}^{H} α_{h}^{(l + 1)} h^{(l)} (x_{i}; u) + {\tilde{f}}_{θ} (x_{i}; α, ε, S) : α_{h}^{(l + 1)} \in R^{d_{y} \times d_{l}}, \\ α \in R^{d_{θ} \times | S |}, S \subseteq_{fin} V [θ, ε]\} . \end{matrix}

Since the set in the first line is a subset of the set in the last line, the equality holds in the above equation. This immediately implies the desired statement from theorem 2.

□

Note

¹

For example, choose the first layer's weight matrix $W^{(1)}$ such that for all $i \in {1, \dots, m}$ ⁠, ${(W^{(1)} x_{i})}_{i} > 0$ and ${(W^{(1)} x_{i})}_{i^{'}} \leq 0$ for all $i^{'} \neq i$ ⁠. This can be achieved by choosing the $i$ th row of $W^{(1)}$ to be $[{(x_{i}^{(raw)})}^{⊤}, ε - 1]$ with $0 < ε \leq δ$ for $i \leq m$ ⁠. Then choose the weight matrices for the $l$ th layer for all $l \geq 2$ such that for all $j$ ⁠, $W_{j, j}^{(l)} \neq 0$ and $W_{j^{'}, j}^{(l)} = 0$ for all $j^{'} \neq j$ ⁠. This guarantees $rank ({[φ (x_{i}; u)]}_{i = 1}^{m}) \geq m$ ⁠.

Acknowledgments

We gratefully acknowledge support from NSF grants 1523767 and 1723381, AFOSR grant FA9550-17-1-0165, ONR grant N00014-18-1-2847, Honda Research, and the MIT-Sensetime Alliance on AI. Any opinions, findings, and conclusions or recommendations expressed in this material are our own and do not necessarily reflect the views of our sponsors.

References

Allen-Zhu

,

Z.

,

Li

,

Y.

, &

Song

,

Z.

(

2018

).

A convergence theory for deep learning via over-parameterization

.

arXiv:1811.03962

.

Arora

,

S.

,

Cohen

,

N.

,

Golowich

,

N.

, &

Hu

,

W.

(

2018

).

A convergence analysis of gradient descent for deep linear neural networks

.

arXiv:1810.02281

.

Arora

,

S.

,

Cohen

,

N.

, &

Hazan

,

E.

(

2018

).

On the optimization of deep networks: Implicit acceleration by overparameterization

. In

Proceedings of the International Conference on Machine Learning

.

Google Scholar

Bartlett

,

P. L.

,

Helmbold

,

D. P.

, &

Long

,

P. M.

(

2019

).

Gradient descent with identity initialization efficiently learns positive-definite linear transformations by deep residual networks

.

Neural Computation

,

31

(

3

),

477

–

502

.

Google Scholar

Bertsekas

,

D. P.

(

1999

).

Nonlinear programming

.

Belmont, MA

:

Athena Scientific

.

Blum

,

A. L.

, &

Rivest

,

R. L.

(

1992

).

Training a 3-node neural network is NP-complete

.

Neural Networks

,

5

(

1

),

117

–

127

.

Google Scholar

Brutzkus

,

A.

, &

Globerson

,

A.

(

2017

).

Globally optimal gradient descent for a convnet with gaussian inputs

. In

Proceedings of the International Conference on Machine Learning

(pp.

605

–

614

).

Google Scholar

Choromanska

,

A.

,

Henaff

,

M.

,

Mathieu

,

M.

,

Ben Arous

,

G.

, &

LeCun

,

Y.

(

2015

).

The loss surfaces of multilayer networks

. In

Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics

(pp.

192

–

204

).

Google Scholar

Davis

,

D.

,

Drusvyatskiy

,

D.

,

Kakade

,

S.

, &

Lee

,

J. D.

(

2019

). Stochastic subgradient method converges on tame functions. In

M.

Overton

(Ed.),

Foundations of computational mathematics

(pp.

1

–

36

).

Berlin

:

Springer

.

Du

,

S. S.

, &

Hu

,

W.

(

2019

).

Width provably matters in optimization for deep linear neural networks

.

arXiv:1901.08572

.

Du

,

S. S.

, &

Lee

,

J. D.

(

2018

).

On the power of over-parameterization in neural networks with quadratic activation

.

arXiv:1803.01206

.

Du

,

S. S.

,

Lee

,

J. D.

,

Li

,

H.

,

Wang

,

L.

, &

Zhai

,

X.

(

2018

).

Gradient descent finds global minima of deep neural networks

.

arXiv:1811.03804

.

Ge

,

R.

,

Lee

,

J. D.

, &

Ma

,

T.

(

2017

).

Learning one-hidden-layer neural networks with landscape design

.

arXiv:1711.00501

.

Goel

,

S.

, &

Klivans

,

A.

(

2017

).

Learning depth-three neural networks in polynomial time

.

arXiv:1709.06010

.

Goodfellow

,

I.

,

Bengio

,

Y.

, &

Courville

,

A.

(

2016

).

Deep learning

.

Cambridge, MA

:

MIT Press

.

Hardt

,

M.

, &

Ma

,

T.

(

2017

).

Identity matters in deep learning

.

arXiv:1611.04231

.

Hardt

,

R. M.

(

1975

).

Stratification of real analytic mappings and images

.

Invent. Math.

,

28

,

193

–

208

.

Google Scholar

He

,

K.

,

Zhang

,

X.

,

Ren

,

S.

, &

Sun

,

J.

(

2016

). Identity mappings in deep residual networks. In

Proceedings of the European Conference on Computer Vision

(pp.

630

–

645

).

Berlin

:

Springer

.

Kakade

,

S. M.

, &

Lee

,

J. D.

(

2018

). Provably correct automatic subdifferentiation for qualified programs. In

S.

Bengio

,

H.

Wallach

,

K.

Grauman

,

N.

Cesa-Bianchi

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

31

(pp.

7125

–

7135

).

Red Hook, NY

:

Curran

.

Kawaguchi

,

K.

(

2016

). Deep learning without poor local minima. In

D.

Lee

,

M.

Sugiyama

,

U.

Luxburg

,

I.

Guyon

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

29

(pp.

586

–

594

).

Red Hook, NY

:

Curran

.

Kawaguchi

,

K.

, &

Bengio

,

Y.

(

2019

).

Depth with nonlinearity creates no bad local minima in ResNets

.

Neural Networks

,

118

,

167

–

174

.

Google Scholar

Kawaguchi

,

K.

,

Huang

,

J.

, &

Kaelbling

,

L. P.

(

2019

).

Effect of depth and width on local minima in deep learning

.

Neural Computation

,

31

(

6

),

1462

–

1498

.

Google Scholar

Kawaguchi

,

K.

,

Xie

,

B.

, &

Song

,

L.

(

2018

).

Deep semi-random features for nonlinear function approximation

. In

Proceedings of the 32nd AAAI Conference on Artificial Intelligence

.

Palo Alto, CA

:

AAAI Press

.

Google Scholar

Laurent

,

T.

, &

Brecht

,

J.

(

2018

).

Deep linear networks with arbitrary loss: All local minima are global

. In

Proceedings of the International Conference on Machine Learning

(pp.

2908

–

2913

).

Google Scholar

Lee

,

J. M.

(

2013

).

Introduction to smooth manifolds

(2nd ed.).

New York

:

Springer

.

Li

,

Y.

, &

Yuan

,

Y.

(

2017

).

Convergence analysis of two-layer neural networks with ReLU activation

. In

I.

Guyon

,

U. V.

Luxburg

,

S.

Bengio

,

N.

Wallach

,

R.

Fergus

,

S.

Vishwanathan

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

30

(pp.

597

–

607

).

Red Hook, NY

:

Curran

.

Google Scholar

Murty

,

K. G.

, &

Kabadi

,

S. N.

(

1987

).

Some NP-complete problems in quadratic and nonlinear programming

.

Mathematical Programming

,

39

(

2

),

117

–

129

.

Google Scholar

Nguyen

,

Q.

, &

Hein

,

M.

(

2017

).

The loss surface of deep and wide neural networks

. In

Proceedings of the International Conference on Machine Learning

(pp.

2603

–

2612

).

Google Scholar

Nguyen

,

Q.

, &

Hein

,

M.

(

2018

).

Optimization landscape and expressivity of deep CNNS

. In

Proceedings of the International Conference on Machine Learning

(pp.

3727

–

3736

).

Google Scholar

Rockafellar

,

R. T.

, &

Wets

,

R. J.-B.

(

2009

).

Variational analysis

.

New York

:

Springer Science & Business Media

.

Saxe

,

A. M.

,

McClelland

,

J. L.

, &

Ganguli

,

S.

(

2014

).

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

.

arXiv:1312.6120

.

Shamir

,

O.

(

2018

).

Are ResNets provably better than linear predictors

? In

S.

Bengio

,

H.

Wallach

,

H.

Larochelle

,

K.

Grauman

,

N.

Cesa-Bianchi

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

31

.

Red Hook, NY

:

Curran

.

Google Scholar

Soltanolkotabi

,

M.

(

2017

). Learning ReLUs via gradient descent. In

I.

Guyon

,

U. V.

Luxburg

,

S.

Bengio

,

H.

Wallach

,

R.

Fergus

,

S.

Vishwanathan

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

30

(pp.

2007

–

2017

).

Red Hook, NY

:

Curran

.

Zhong

,

K.

,

Song

,

Z.

,

Jain

,

P.

,

Bartlett

,

P. L.

, &

Dhillon

,

I. S.

(

2017

).

Recovery guarantees for one-hidden-layer neural networks

. In

Proceedings of the International Conference on Machine Learning

(pp.

4140

–

4149

).

Google Scholar

Zou

,

D.

,

Cao

,

Y.

,

Zhou

,

D.

, &

Gu

,

Q.

(

2018

).

Stochastic gradient descent optimizes over-parameterized deep ReLU networks

.

arXiv:1811.08888

.

2019

Massachusetts Institute of Technology. Published under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.

Every Local Minimum Value Is the Global Minimum Value of Induced Model in Nonconvex Machine Learning

Abstract

1 Introduction

2 Preliminaries

2.1 Problem Description

2.2 Notation

3 Nonconvex Optimization Landscapes for Machine Learning

3.1 Assumptions

3.2 Theory for Critical Points

3.2.1 Geometric View

3.2.2 Examples

3.3 Theory for Local Minima

3.3.1 Geometric View

4 Applications to Deep Neural Networks

4.1 Example: ResNets

4.1.1 Background

4.2 Result

4.2.1 Example: Deep Nonlinear Networks with Locally Induced Partial Linear Structures

4.2.2 Background

4.2.3 Result

4.3 Proof Idea in Applications of Theorem 2

5 Conclusion

Appendix: Proofs of Theoretical Results

A.1 Proof of Theorem 1

A.2 Proof of Theorem 2

A.3 Proof of Lemma 1

A.4 Proof of Theorem 3

A.5 Proof of Theorem 4

Note

Acknowledgments

References

Email alerts

Cited By

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

Every Local Minimum Value Is the Global Minimum Value of Induced Model in Nonconvex Machine Learning

Abstract

1 Introduction

2 Preliminaries

2.1 Problem Description

2.2 Notation

3 Nonconvex Optimization Landscapes for Machine Learning

3.1 Assumptions

3.2 Theory for Critical Points

3.2.1 Geometric View

3.2.2 Examples

3.3 Theory for Local Minima

3.3.1 Geometric View

4 Applications to Deep Neural Networks

4.1 Example: ResNets

4.1.1 Background

4.2 Result

4.2.1 Example: Deep Nonlinear Networks with Locally Induced Partial Linear Structures

4.2.2 Background

4.2.3 Result

4.3 Proof Idea in Applications of Theorem 2

5 Conclusion

Appendix: Proofs of Theoretical Results

A.1 Proof of Theorem 1

A.2 Proof of Theorem 2

A.3 Proof of Lemma 1

A.4 Proof of Theorem 3

A.5 Proof of Theorem 4

Note

Acknowledgments

References

Email alerts

Cited By

Related Articles

Related Book Chapters

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

This Feature Is Available To Subscribers Only