Center Manifold Analysis of Plateau Phenomena Caused by Degeneration of Three-Layer Perceptron

Tsutsui, Daiji

doi:10.1162/neco_a_01268

Abstract

A hierarchical neural network usually has many singular regions in the parameter space due to the degeneration of hidden units. Here, we focus on a three-layer perceptron, which has one-dimensional singular regions comprising both attractive and repulsive parts. Such a singular region is often called a Milnor-like attractor. It is empirically known that in the vicinity of a Milnor-like attractor, several parameters converge much faster than the rest and that the dynamics can be reduced to smaller-dimensional ones. Here we give a rigorous proof for this phenomenon based on a center manifold theory. As an application, we analyze the reduced dynamics near the Milnor-like attractor and study the stochastic effects of the online learning.

1 Introduction

A three-layer perceptron is one of the simplest hierarchical learning machines. Mathematically, it is defined by the function

\begin{matrix} f_{(d)} (x; θ) = \sum_{i = 1}^{d} v_{i} ϕ (w_{i} \cdot x + b_{i}), x \in R^{n}, \\ θ = (w_{1}, \dots, w_{d}, b_{1}, \dots, b_{d}, v_{1}, \dots, v_{d}), \end{matrix}

(1.1)

where

θ

is a system parameter with

w_{1}, \dots, w_{d} \in R^{n}

being the weight vectors for the second layer,

b_{1}, \dots, b_{d} \in R

the bias terms for the second,

v_{1}, \dots, v_{d} \in R^{m}

the weight vectors for the third, and

ϕ

an activation function. Throughout this article, we assume that the activation function is twice differentiable. Figure 1 is a schematic diagram of the three-layer perceptron. We shall call the function an (⁠

n

-

d

-

m

⁠)-perceptron. The numbers

n

and

m

are fixed at the outset as the sizes of input and output vectors, while the number

d

of hidden units can be varied in our analysis. For notational simplicity, we incorporate the bias

b

in the weight

w

as

w = (b, w^{1}, \dots, w^{n})

⁠, and accordingly, we enlarge

x

as

x = (1, x_{1}, \dots, x_{n})

⁠. By using these conventions, we obtain the abridged presentation of the three-layer perceptron as

f_{(d)} (x; θ) = \sum_{i = 1}^{d} v_{i} ϕ (w_{i} \cdot x) .

(1.2)

Figure 1:

View large Download slide

A schematic diagram of a three-layer perceptron presented in equation 1.1.

In this article, we treat the supervised learning. The goal of the supervised learning is to find an optimal parameter $θ$ so that $f_{(d)} (x; θ)$ approximates a given target function $T (x)$ ⁠. Such a problem is based on the universal approximation property of the three-layer perceptron. For a suitable activation function $ϕ$ (e.g., sigmoidal or ReLU), the model, equation 1.2, can approximate quite a wide range of functions as the number $d$ of hidden units tends to infinity (Cybenko, 1989; Funahashi, 1989; Sonoda & Murata, 2017).

The (averaged) gradient descent method is a standard method for finding an optimal value of

θ

numerically. Suppose that a loss function

ℓ (x, y)

is nonnegative and is equal to zero if and only if

y = T (x)

(e.g., the squared error

| | y - {T (x) | |}^{2}

⁠). In the gradient descent method, we aim at minimizing

L_{(d)} (θ) : = E_{x} [ℓ (x, f_{(d)} (x; θ))]

(1.3)

by changing the parameter

θ

iteratively as

θ_{t + 1} = θ_{t} - ɛ \frac{\partial L_{(d)}}{\partial θ} (θ_{t}),

(1.4)

where

ɛ > 0

is a learning constant. Here, we assume that the input vector

x

is a random variable drawn according to an unknown probability distribution and

E_{x}

denotes the expectation with respect to

x

⁠. In order for the differential

\partial L_{(d)} / \partial θ

to make sense, we also assume that

ℓ (x, y)

is differentiable with respect to

y

and that we can interchange the order of the differentiation

\partial / \partial θ

and the expectation

E_{x}

⁠. We study the dynamical system, which represents the averaged gradient descent method with infinitesimal learning constant:

\begin{matrix} \frac{d θ}{d t} = - \frac{\partial L_{(d)}}{\partial θ} (θ) . \end{matrix}

(1.5)

The parameter

θ

descends along the gradient of

L_{(d)}

into a local minimum. In practice, the expectation in equation is replaced with the arithmetic mean over a given data set, or, roughly, with a single realization of the random variable

ℓ (x, f_{(d)} (x; θ))

for each learning iteration. Such a learning method involving some stochastic effects is called a stochastic gradient descent method.

Fukumizu and Amari (2000) studied singular regions arising from degeneration of hidden units of a three-layer perceptron. Here, the degeneration of hidden units means that several weight parameters $w_{i}$ take the same value and the effective number of hidden units becomes fewer than $d$ ⁠. When $m = 1$ ⁠, they found a novel type of singular region, often called a Milnor-like attractor. This region has both an attractive part consisting of local minima of $L_{(d)}$ and a repulsive part consisting of saddle points. In practical learning, there may be some stochastic effects. Therefore, once the parameter $θ$ is trapped in the attractive part of this region, it fluctuates in the region by stochastic effects for a long time, until it reaches the repulsive part. This may cause serious stagnation of learning, called plateau phenomena. Later, Amari, Ozeki, Karakida, Yoshida, and Okada (2018) pointed out a notable fact that a Milnor-like attractor may not cause serious stagnation of learning when $m \geq 2$ ⁠, which is also treated in this article.

More quantitative analyses for

m = 1

have also been carried out by Coussear, Ozeki, and Amari (2008), Wei, Zhang, Cousseau, Ozeki, and Amari (2008), and Amari et al. (2018) in the simplest case

d = 2

⁠. In particular, Wei et al. (2008) introduced a new coordinate system in the parameter space by

\begin{matrix} \{\begin{matrix} w = \frac{v_{1} w_{1} + v_{2} w_{2}}{v_{1} + v_{2}} \\ v = v_{1} + v_{2} \\ u = w_{1} - w_{2} \\ z = \frac{v_{1} - v_{2}}{v_{1} + v_{2}} \end{matrix}, \end{matrix}

(1.6)

and claimed, based on evidence observed in numerical simulations, that when the initial point is taken near a Milnor-like attractor, the parameters

(w, v)

quickly converge to equilibrium values

(w^{*}, v^{*})

⁠. They hypothesized that this would always be the case and analyzed only the reduced dynamical system for the subparameters

(u, z)

⁠, setting the remaining parameters

(w, v)

to be

(w^{*}, v^{*})

⁠. However, to the best of our knowledge, no mathematical justification for this hypothesis has been established.

The objective of this article is to provide a solid ground on Amari et al.'s (2018) point of view. We introduce a new coordinate system that admits a center manifold structure around a special point on the Milnor-like attractor. By using the coordinate system, we can analyze the Milnor-like attractor more rigorously and integrate the reduced dynamical system explicitly to obtain analytical trajectories. The obtained trajectories are comparable to the preceding work. It is confirmed by several settings of numerical simulations that trajectories in actual learning agree with the analytical ones.

In addition to the averaged gradient descent method, we also address online learning, a stochastic gradient descent method. Around a Milnor-like attractor, the behavior of sample paths by the online learning seems qualitatively different from that of trajectories by the averaged gradient descent. To investigate why they are different, we divide the dynamics of parameters into fast and slow ones, as is the case in the averaged gradient descent. In this case, we observed in numerical simulations that the fast parameters fluctuate intensively around the center manifold for the averaged system. We show that such a deviation of the fast parameter from the center manifold can influence a trend of the slow parameter.

This article is organized as follows. In section 2, we give a quick review of Amari et al.'s (2018) work. In section 3, after a brief account of the center manifold theory, we introduce a new coordinate system in the parameter space and prove that it admits the center manifold structure. In section 4, we carry out numerical simulations and observe the center manifold structure around a Milnor-like attractor. In section 5, we consider the online learning from the viewpoint of the center manifold theory. Section 6 offers concluding remarks.

2 Singular Region and Milnor-Like Attractor

In this section, we give a quick review of the Milnor-like attractor that Fukumizu and Amari (2000) found, which appears when the number $m$ of output units is equal to 1. We also mention an interesting insight by Amari et al. (2018) for the case $m \geq 2$ ⁠.

The parameter space of a perceptron is sometimes called a perceptron manifold. However, in many cases, it is not really a manifold since it usually contains a subset whose points correspond to the same input-output relation. Such a subset is usually referred to as a singular region. In general, there are many singular regions due to the degeneration of hidden units. For example, let us consider an (⁠

n

-2-

m

⁠)-perceptron. Then the subset

R (w, v) : = {θ = (w_{1}, w_{2}, v_{1}, v_{2}) | w_{1} = w_{2} = w, v_{1} + v_{2} = v}

of the parameter space forms a typical singular region. In fact, on the subset

R (w, v)

⁠, an (⁠

n

-2-

m

⁠)-perceptron

f_{(2)} (x; θ)

is reduced to the following (⁠

n

-1-

m

⁠)-perceptron:

f_{(1)} (x; w, v) : = v ϕ (w \cdot x) .

On such a singular region, some properties of $L_{(1)}$ are inherited by $L_{(2)}$ ⁠. The following lemma implies that a criticality is a hereditary property.

Lemma 1.

Let $θ^{*} = (w^{*}, v^{*})$ be a critical point of $L_{(1)}$ ⁠. Then the parameter $θ = (w_{1}, w_{2}, v_{1}, v_{2}) = (w^{*}, w^{*}, λ v^{*}, (1 - λ) v^{*})$ is a critical point of $L_{(2)}$ for any $λ \in R$ ⁠.

Proof.

\begin{matrix} \frac{\partial L_{(2)}}{\partial w_{i}} (θ) & = & E [(\frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y} \cdot v_{i}) ϕ^{'} (w^{*} \cdot x) x^{T}] = λ_{i} \frac{\partial L_{(1)}}{\partial w} (θ^{*}), \\ \frac{\partial L_{(2)}}{\partial v_{i}} (θ) & = & E [\frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y} ϕ (w^{*} \cdot x)] = \frac{\partial L_{(1)}}{\partial v} (θ^{*}), i = 1, 2, \end{matrix}

where

λ_{1} : = λ

and

λ_{2} : = 1 - λ

⁠. Since

θ^{*}

is a critical point of

L_{(1)}

⁠, these are all zero.

When $m = 1$ ⁠, in particular, every point $θ \in R (w^{*}, v^{*})$ is a critical point of $L_{(2)}$ ⁠, since the parameter $v$ ⁠, as well as the output $f^{(d)} (x; θ)$ ⁠, is a scalar quantity. In this case, the second-order property of $L_{(1)}$ is also inherited by $L_{(2)}$ to some extent, and the singular region $R (w^{*}, v^{*})$ may have an interesting structure, which causes serious stagnation of learning.

Proposition 1

(Fukumizu & Amari, 2000). Let

m = 1

and

θ^{*} = (w^{*}, v^{*})

be a strict local minimizer of

L_{(1)}

with

v^{*} \neq 0

⁠. Define an

(n + 1) \times (n + 1)

matrix

H

by

H : = E_{x} [\frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y} v^{*} ϕ^{''} (w^{*} \cdot x) x x^{T}],

(2.1)

and for

λ \in R

⁠, let

θ_{λ} : = (w^{*}, w^{*}, λ v^{*}, (1 - λ) v^{*}) .

If the matrix

H

is positive (resp. negative) definite, then the point

θ = θ_{λ}

is a local minimizer (resp. saddle point) of

L_{(2)}

for any

λ \in (0, 1)

and is a saddle point (resp. local minimizer) for any

λ \in R ∖ [0, 1]

⁠. On the other hand, if the matrix

H

is indefinite, then the point

θ_{λ}

is a saddle point of

L_{(2)}

for all

λ \in R ∖ {0, 1}

⁠.

This proposition implies that the one-dimensional region $R (w^{*}, v^{*}) = {θ_{λ} ∣ λ \in R}$ may have both attractive parts and repulsive parts in the gradient descent method. Such a region is referred to as a Milnor-like attractor (Wei et al., 2008). The parameter $θ$ near the attractive part flows into the Milnor-like attractor and fluctuates in the region for a long time, until it reaches the repulsive part.

The original theorem (Fukumizu & Amari, 2000) is for an (⁠ $n$ - $d$ -1)-perceptron that contains (⁠ $n$ -(⁠ $d$ -1)-1)-perceptron as a subnetwork and that the phenomenon itself is universal with respect to the number $d$ of hidden units. The proposition above for (⁠ $n$ -2-1)-perceptron is a minimal version.

We also remark that the point $θ_{λ}$ cannot be a strict local minimizer since $L_{(2)}$ takes the same value on the singular region ${θ_{λ} ∣ λ \in R}$ and is flat along its direction. The proof of proposition 1 is given mainly by a discussion of the Hessian matrix of $L_{(2)}$ ⁠; however, we need to treat higher-order derivatives of $L_{(2)}$ than the second order, since the Hessian matrix degenerates on the singular region (see appendix A).

Let us suppose a situation where a three-layer perceptron has some redundant hidden units to represent the target function

T (x)

⁠. Mathematically, we suppose that a true parameter

θ_{t r u e}

exists (⁠

T (x) = f_{(2)} (x; θ_{t r u e})

⁠) and that it lies in the singular region

R (w^{*}, v^{*})

⁠. In this case, the function

L_{(2)}

takes the same value

L_{(1)} (w^{*}, v^{*}) = 0

on

R (w^{*}, v^{*})

⁠. Therefore, every point of

R (w^{*}, v^{*})

becomes a global minimizer of

L_{(2)}

⁠, and a Milnor-like attractor does not appear. In fact, one can check that the assumption of proposition 1 fails as follows. For each

x \in R^{n}

⁠, we obtain

\partial ℓ (x, f_{(1)} (x; w^{*}, v^{*})) / \partial y = 0,

since a loss function

ℓ (x, y)

takes its minimum 0 at

y = T (x) = f_{(1)} (x; w^{*}, v^{*})

⁠. This implies that the matrix

H

becomes the zero matrix. Thus,

H

is neither positive nor negative definite.

We next treat the case when $m \geq 2$ ⁠. There also exists a one-dimensional region consisting of critical points due to lemma 1. However, in this case, the region becomes simply repulsive and does not have an attractive part, as the following theorem asserts.

Theorem 1.

Let

θ^{*}

=

(w^{*}, v^{*})

be a local minimizer of

L_{(1)}

⁠. If the

m \times (n + 1)

matrix

E_{x} [\frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}]

(2.2)

is nonzero, then

θ_{λ} = (w^{*}, w^{*}, λ v^{*}, (1 - λ) v^{*})

is a saddle point of

L_{(2)}

for any

λ \in R

⁠, where we regard the derivative

\partial ℓ / \partial y

as a column vector.

Amari et al. (2018) stated a prototype of theorem 1, although they did not give a full proof. In fact, we found that some additional assumption was necessary to prove their assertion. In theorem 1, we have added a mild assumption that the matrix, equation , is nonzero. Note that since

θ^{*} = (w^{*}, v^{*})

is a local minimizer of

L_{(1)}

⁠, it holds that

0 = \frac{\partial L_{(1)}}{\partial w} (θ^{*}) = {(v^{*})}^{T} E_{x} [\frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}] .

Thus, the matrix has a kernel whose dimension is greater than or equal to one. Hence, the assumption automatically fails when

m = 1

⁠. This is an underlying mechanism for proposition 1.

3 Center Manifold of Milnor-Like Attractor

In their analysis of an (⁠

n

-2-1)-perceptron, Wei et al. (2008) introduced a coordinate transformation,

\begin{matrix} \{\begin{matrix} w = \frac{v_{1} w_{1} + v_{2} w_{2}}{v_{1} + v_{2}} \\ v = v_{1} + v_{2} \\ u = w_{1} - w_{2} \\ z = \frac{v_{1} - v_{2}}{v_{1} + v_{2}} \end{matrix}, \end{matrix}

(3.1)

and claimed that the parameters

(w, v)

quickly converge to

(w^{*}, v^{*})

when the initial point is taken near a Milnor-like attractor. Amari et al. (2018) mentioned that the dynamics in this coordinate system should be analyzed by using the center manifold theory, and they analyzed only the reduced dynamical system for the subparameters

(u, z)

⁠, setting the remaining parameters

(w, v)

to be

(w^{*}, v^{*})

⁠. Strictly speaking, however, their coordinate system does not admit any center manifold structure, and their claim is at the stage of hypothesis.

In this section, we give a rigorous justification for their hypothesis. We first give a quick review of the center manifold theory and then introduce a new coordinate system under which the center manifold structures arise near certain points on the Milnor-like attractor.

3.1 Brief Review of Center Manifold

Suppose that we are given a dynamical system,

\{\begin{matrix} \dot{x} (t) = A x (t) + f (x (t), y (t)) \\ \dot{y} (t) = B y (t) + g (x (t), y (t)) \end{matrix},

(3.2)

for the parameters

(x, y) \in R^{d_{1}} \times R^{d_{2}}

⁠, where

A

and

B

are constant matrices and

f

and

g

are

C^{2}

functions such that they, along with their first derivatives, vanish at the origin. We assume that all the eigenvalues of

A

have zero real parts, while all the eigenvalues of B have negative real parts. This assumption means that parameter

y

converges to the origin exponentially quickly, and the parameter

x

is driven only by the higher-order terms of

f

and evolves very slowly compared with

y

⁠. Since

f

and

g

are of the second order with respect to

x

and

y

⁠, the assumptions for equation imply that the coefficient matrix of the linearization of the system has the form

(\begin{matrix} A & O \\ O & B \end{matrix}) .

Definition 1.

A set $S \subset R^{d_{1}} \times R^{d_{2}}$ is said to be a local invariant manifold of equation 3.2 if for $(x_{0}, y_{0}) \in S$ ⁠, the solution $(x (t), y (t))$ of equation 3.2 with $(x (0), y (0)) = (x_{0}, y_{0})$ is in $S$ for $| t | < T$ with some $T > 0$ ⁠.

Definition 2.

A local invariant manifold represented in the form of $y = h (x)$ is called a local center manifold (or simply a center manifold) if $h$ is differentiable and satisfies $h (0) = 0$ and $\frac{\partial h}{\partial x} (0) = O$ ⁠.

The following center manifold theorems give us a method of simplifying a dynamical system around an equilibrium point.

Proposition 2

(Center manifold theorem 1: Carr, 1981). Equation 3.2 has a center manifold $y = h (x)$ for $| | x | | < δ$ ⁠, for some $δ > 0$ and $C^{2}$ function $h$ ⁠.

Proposition 3

(Center manifold theorem 2: Carr, 1981). Suppose that the origin

u = 0

is a stable equilibrium point of the reduced dynamical system

\dot{u} (t) = A u (t) + f (u (t), h (u (t))) .

(3.3)

Let

(x (t), y (t))

be a solution of equation with the initial value

(x_{0}, y_{0})

⁠. Then if

| | (x_{0}, y_{0}) | |

is sufficiently small, there exists a solution

u (t)

of equation such that

\begin{matrix} x (t) & = & u (t) + O (e^{- γ t}), \\ y (t) & = & h (u (t)) + O (e^{- γ t}), \end{matrix}

as

t \to \infty

⁠, where

γ

is a positive constant.

Proposition 3 asserts that the parameter $(x, y)$ approaches the center manifold $y = h (x)$ quickly and then evolves along it. Thus, the dynamical system, equation 3.2, around the origin is essentially controlled by the slow parameter $x$ and reduced to the lower-dimensional system.

3.2 Main Results

Let us return to the analysis of an (⁠

n

-2-1)-perceptron. In a column vector representation, the dynamical system, equation , for the (⁠

n

-2-1)-perceptron is written as

\dot{θ} = - {(\frac{\partial L_{(2)}}{\partial θ} (θ))}^{T} .

We consider another coordinate system

ξ = ξ (θ)

and investigate the dynamical system in it. By the coordinate transformation, the dynamical system above is transformed to

\begin{matrix} \dot{ξ} = - \frac{\partial ξ}{\partial θ} {(\frac{\partial ξ}{\partial θ})}^{T} {(\frac{\partial L_{(2)}}{\partial ξ} (ξ))}^{T} . \end{matrix}

(3.4)

Thus, the coefficient matrix of its linearization at a critical point

ξ = ξ^{*}

is

\begin{matrix} \frac{\partial \dot{ξ}}{\partial ξ} (ξ^{*}) & = & - {\frac{\partial}{\partial ξ} \{\frac{\partial ξ}{\partial θ} {(\frac{\partial ξ}{\partial θ})}^{T} {(\frac{\partial L_{(2)}}{\partial ξ} (ξ))}^{T}\}|}_{ξ = ξ^{*}} \\ = & - \frac{\partial ξ}{\partial θ} {(\frac{\partial ξ}{\partial θ})}^{T} \frac{\partial^{2} L_{(2)}}{\partial ξ \partial ξ} (ξ^{*}), \end{matrix}

where we used

(\partial L_{(2)} / \partial ξ) (ξ^{*}) = 0

⁠. This relation implies that the coefficient matrix has the same rank as the Hessian matrix

(\partial^{2} L_{(2)} / \partial ξ \partial ξ) (ξ^{*})

⁠. In particular, the rank of the coefficient matrix of the linearization does not depend on the choice of a coordinate system.

We focus on the dynamics of the learning process around the two points, $θ = θ_{0}, θ_{1}$ ⁠, which are boundaries of the repulsive and attractive parts of a Milnor-like attractor ${θ_{λ} | λ \in R}$ ⁠. Concretely, they are denoted as $θ_{0} = (w^{*}, w^{*}, 0, v^{*})$ and $θ_{1} = (w^{*}, w^{*}, v^{*}, 0)$ ⁠, where $(w^{*}, v^{*})$ is a minimizer of the loss $L_{(1)}$ for the (⁠ $n$ -1-1)-perceptron as mentioned in proposition 1. This is because the rank of the Hessian matrix at $θ_{λ}$ degenerates by one dimension for $λ \neq 0, 1$ and by $n + 2$ dimension for $λ = 0, 1$ ⁠, which is shown in appendix A.

We introduce a new coordinate system

ξ = (w, v, u, z)

by

\{\begin{matrix} w = \frac{v_{1} (w_{1} - w^{*}) + v_{2} (w_{2} - w^{*})}{v^{*}} + w^{*} \\ v = v_{1} + v_{2} \\ u = \frac{v_{2} (w_{1} - w^{*}) - v_{1} (w_{2} - w^{*})}{v^{*}} \\ z = v_{1} - v_{2} \end{matrix}

(3.5)

This formula defines a coordinate system on the region

{v_{1}^{2} + v_{2}^{2} \neq 0}

⁠. Under the coordinate system, equation , the critical points

θ_{λ}

are denoted as

ξ_{λ} = (w^{*}, v^{*}, 0, (2 λ - 1) v^{*}) .

In particular,

ξ_{0} = (w^{*}, v^{*}, 0, - v^{*})

and

ξ_{1} = (w^{*}, v^{*}, 0, v^{*})

⁠. Now we arrive at the main theorem of this article:

Theorem 2.

In the coordinate system $ξ = (w, v, u, z)$ ⁠, the dynamical system, equation 1.5, admits a center manifold structure around the critical points $θ = θ_{0}, θ_{1}$ in which $(w, v)$ converge exponentially fast.

To prove the theorem, we make use of the following lemma.

Lemma 2.

If the matrix $X$ is positive definite and $Y$ is positive semidefinite, all the eigenvalues of the matrix $X Y$ are nonnegative.

Proof.

The matrix

X Y

is rewritten as

X Y = X^{\frac{1}{2}} (X^{\frac{1}{2}} Y X^{\frac{1}{2}}) X^{- \frac{1}{2}},

where

X^{\frac{1}{2}}

is a unique positive-definite matrix such that

{(X^{\frac{1}{2}})}^{2} = X

⁠. Here, the matrix

Z : = X^{\frac{1}{2}} Y X^{\frac{1}{2}}

is positive semidefinite. Hence, for each eigenvector

a

of

Z

⁠, the vector

X^{\frac{1}{2}} a

is an eigenvector of the matrix

X Y

⁠, and the corresponding eigenvalue is nonnegative.

Proof of Theorem 2.

The proof is essentially based on a straightforward calculation. The coefficient matrix of the linearization of the dynamical system, equation , under the coordinate system, equation , splits into

(w, v)

part and

(u, z)

part for

λ = 0, 1

⁠. In fact, for

λ \in R

⁠, the negative of the coefficient matrix is written as

\begin{matrix} \begin{matrix} \overset{w, v}{\overset{⏞}{}} \overset{u}{\overset{⏞}{}} \overset{z}{\overset{⏞}{}} \end{matrix} \\ {\tilde{A}}_{λ} & = & \begin{matrix} w, v \{ \\ u \{ \\ z \{ \end{matrix} (\begin{matrix} (1 + 2 k_{λ}) Q + \frac{1 + 3 k_{λ}}{1 + 2 k_{λ}} H & (1 + 2 k_{λ}) P & - \frac{(- 1 + 2 λ) k_{λ}}{1 + 2 k_{λ}} H & 0 \\ 2 P^{T} & 2 R & 0 & 0 \\ - \frac{(- 1 + 2 λ) k_{λ}}{1 + 2 k_{λ}} H & 0 & - \frac{k_{λ}}{1 + 2 k_{λ}} H & 0 \\ 0 & 0 & 0 & 0 \end{matrix}), \end{matrix}

and the system is written as

\dot{ξ} = - {\tilde{A}}_{λ} (ξ - ξ_{λ}) + {\tilde{g}}_{λ} (ξ),

where

{\tilde{g}}_{λ}

is the higher-order term, which vanish at the

ξ = ξ_{λ}

together with its first derivative. Here,

\begin{matrix} k_{λ} & : = & (1 - λ) λ, \\ P & : = & E_{x} [(\partial^{2} ℓ) v^{*} ϕ (w^{*} \cdot x) ϕ^{'} (w^{*} \cdot x) x], \\ Q & : = & E_{x} [(\partial^{2} ℓ) {(v^{*})}^{2} ϕ^{'} {(w^{*} \cdot x)}^{2} x x^{T}], \\ R & : = & E_{x} [(\partial^{2} ℓ) ϕ {(w^{*} \cdot x)}^{2}], \\ \partial ℓ & : = & \frac{\partial ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y}, \\ \partial^{2} ℓ & : = & \frac{\partial^{2} ℓ (x, f_{(1)} (x; θ^{*}))}{\partial y^{2}}, \end{matrix}

and

H

is the matrix defined by equation .

H

and

Q

are matrices,

P

is a column vector, and

R

is a scalar. For

λ = 0

or 1, the negative of the coefficient matrix is of the form

\begin{matrix} \overset{w, v}{\overset{⏞}{}} \overset{u}{\overset{⏞}{}} \overset{z}{\overset{⏞}{}} \\ {\tilde{A}}_{0} = {\tilde{A}}_{1} & = & \begin{matrix} w, v { \\ u { \\ z { \end{matrix} (\begin{matrix} Q + H & P & 0 & 0 \\ 2 P^{T} & 2 R & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix}) . \end{matrix}

(3.6)

We show that all the eigenvalues of

(w, v)

-block of the coefficient matrix

- {\tilde{A}}_{0}

are strictly negative. Recall that the coefficient matrix at a critical point

ξ^{*}

of the dynamical system, equation , is given by

- \frac{\partial ξ}{\partial θ} {(\frac{\partial ξ}{\partial θ})}^{T} \frac{\partial^{2} L_{(2)}}{\partial ξ \partial ξ} (ξ^{*}) .

Applying lemma 2 to

X = (\partial ξ / \partial θ) {(\partial ξ / \partial θ)}^{T}

and

Y = (\partial^{2} L_{(2)} / \partial ξ \partial ξ) (ξ_{0})

⁠, all the eigenvalues of the coefficient matrix

- {\tilde{A}}_{0}

are nonpositive. One can see that the Hessian matrix is positive semidefinite and degenerates by

n + 2

dimension at

θ = θ_{0}, θ_{1}

in appendix A. Since a coordinate transformation preserves the rank of the coefficient matrix of linearization,

{\tilde{A}}_{0}

degenerates by

n + 2

dimension, which is equal to the size of

(u, z)

-block. This implies that the

(w, v)

-block is of full rank, and thus all the eigenvalues of

(w, v)

-block are strictly negative. It is proved similarly for

ξ = ξ_{1}

⁠.

Due to proposition 2, there are center manifolds parametrized by $(u, z)$ around $θ = θ_{0}, θ_{1}$ respectively.

3.3 Reduced Dynamical System

By virtue of proposition 3 and theorem 2, we can assume that the dynamics of the gradient descent is on the center manifold near the points

θ = θ_{0}, θ_{1}

⁠. Thus, we can reduce the dynamical system into that of

(u, z)

⁠. Recalling the coefficient matrix, equation , we can see that

\dot{u}

and

\dot{z}

have no first-order terms. In more detail, calculating the Taylor expansion of

(\dot{u}, \dot{z})

up to the second order around

ξ = ξ_{1}

⁠, we obtain

\begin{matrix} \dot{u} = \frac{1}{v^{*}} {- (P \cdot (w - w^{*})) (u + (w - w^{*})) \\ - (v - v^{*}) (R I + \frac{1}{2} H) (u + (w - w^{*})) \\ + \frac{1}{2} (z - v^{*}) H (u + (w - w^{*}))} + O (| | ξ - ξ_{1} {| |}^{3}), \end{matrix}

(3.7)

\begin{matrix} \dot{z} & = & \frac{1}{v^{*}} \{- {(w - w^{*})}^{T} Q (u + (w - w^{*})) \\ - (v - v^{*}) (P \cdot (u + (w - w^{*}))) \\ - \frac{1}{2} {(w - w^{*})}^{T} H (w - w^{*}) + \frac{1}{2} u^{T} H u\} + O (| | ξ - ξ_{1} {| |}^{3}), \end{matrix}

(3.8)

where

I

denotes the

(n + 1) \times (n + 1)

identity matrix. Now we consider the reduced dynamical system on the center manifold. Here, the center manifold

(w, v) = h (u, z)

satisfies that

h (u, z) = [\begin{matrix} w (u, z) \\ v (u, z) \end{matrix}] = [\begin{matrix} w^{*} \\ v^{*} \end{matrix}] + O (∥ u, z - v^{*} ∥^{2}),

by definition. This gives an approximation of the dynamics on the center manifold near

ξ = ξ_{1}

as

\begin{matrix} \dot{u} & = & \frac{1}{2 v^{*}} (z - v^{*}) H u + O (∥ u, z - v^{*} ∥^{3}), \\ \dot{z} & = & \frac{1}{2 v^{*}} u^{T} H u + O (∥ u, z - v^{*} ∥^{3}) . \end{matrix}

(3.9)

Neglecting the higher-order terms, we can integrate this equation to obtain

{∥ u ∥}^{2} = {(z - v^{*})}^{2} + C,

(3.10)

where

C

is an integral constant.

Around the point

ξ = ξ_{0}

⁠, we obtain the similar dynamics,

\begin{matrix} \dot{u} & = & - \frac{1}{2 v^{*}} (z + v^{*}) H u + O (∥ u, z + v^{*} ∥^{3}), \\ \dot{z} & = & - \frac{1}{2 v^{*}} u^{T} H u + O (∥ u, z + v^{*} ∥^{3}), \end{matrix}

and the relation,

{∥ u ∥}^{2} = {(z + v^{*})}^{2} + C .

We remark that theorem 2 is valid even when there exists a true parameter in the singular region $R (w^{*}, v^{*})$ ⁠; however, in this case, such a simple form of the reduced dynamical system as equation 3.9 is not obtained. As mentioned above, this case implies that $H$ becomes the zero matrix. Then, the second-order terms of the reduced dynamical system, equation 3.9, vanish, and the third-order terms become dominant. Thus, we have to take into account the cross terms between $(w - w^{*}, v - v^{*})$ and $(u, z - v^{*})$ ⁠. It needs to calculate the center manifold $(w, v) = h (u, z)$ up to the second order, which makes the analysis complicated.

Finally, we remark on a difference between our analysis and previous work. Wei et al. (2008) have studied a reduced dynamical system in the vicinity of the whole part of a Milnor-like attractor. On the other hand, a center manifold is defined locally, and center manifolds around each of two points cannot be connected at a midpoint in general. Thus, one cannot discuss a center manifold defined around the entire region of a Milnor-like attractor.

3.4 More General Models

Our results can be extended to a more general model including multilayer perceptrons whose output is one-dimensional. In this section, we consider a parameterized family of functions that can be written as

\begin{matrix} f (x; θ) & : = & g (x, τ) + v_{1} φ (x; w_{1}, τ) + v_{2} φ (x; w_{2}, τ), \\ θ & = & (w_{1}, w_{2}, v_{1}, v_{2}, τ), \end{matrix}

(3.11)

where we assume that

g (x; τ)

and

φ (x; w, τ)

are twice differentiable with respect to

τ

and

(w, τ)

⁠, respectively.

A multilayer perceptron

{\tilde{α}}^{(L)} (x; θ)

with

L

layers defined recursively as

\begin{matrix} α^{(0)} (x; θ) & : = & {(1, x_{1}, \dots, x_{n_{0}})}^{T}, \\ {\tilde{α}}^{(ℓ)} (x; θ) & : = & W^{(ℓ)} α^{(ℓ - 1)} (x; θ), \\ α^{(ℓ)} (x; θ) & : = & {(1, ϕ ({\tilde{α}}_{1}^{(ℓ)} (x; θ)), \dots, ϕ ({\tilde{α}}_{n_{ℓ}}^{(ℓ)} (x; θ)))}^{T}, 1 \leq ℓ \leq L, \\ θ & = & (W^{(1)}, \dots, W^{(L)}), \end{matrix}

for each

x = {(x_{1}, \dots, x_{n_{0}})}^{T} \in R^{n_{0}}

⁠, where

W^{(ℓ)}

is an

n_{ℓ} \times (n_{ℓ - 1} + 1)

matrix,

n_{0}, n_{1}, \dots, n_{L} \in N

⁠, and

ϕ

is a twice-differentiable activation function. Assuming that the dimension

n_{L}

of the output is equal to 1 and denoting

\begin{matrix} W^{(L - 1)} = {[\begin{matrix} w_{1} & w_{2} & \dots & w_{n_{L - 1}} \end{matrix}]}^{T}, \\ W^{(L)} = [\begin{matrix} v_{0} & v_{1} & v_{2} & \dots & v_{n_{L - 1}} \end{matrix}], \end{matrix}

a multilayer perceptron is represented as model by letting

\begin{matrix} φ (x; w, τ) & : = & ϕ (w \cdot α^{(L - 2)} (x; τ)), \\ g (x; τ) & : = & \sum_{i = 3}^{n_{L - 1}} v_{i} ϕ (w_{i} \cdot α^{(L - 2)} (x; τ)), \\ τ & : = & (W^{(1)}, \dots, W^{(L - 2)}, w_{3}, \dots, w_{n_{L - 1}}, v_{0}, v_{3}, \dots, v_{n_{L - 1}}) . \end{matrix}

Our main result is extended to the model, equation as follows. Let

θ^{*} = (w^{*}, v^{*}, τ^{*})

be a strict local minimizer of an averaged loss function for the degenerate model:

f_{(1)} (x; \tilde{w}, \tilde{v}, τ) = g (x; τ) + \tilde{v} φ (x; \tilde{w}, τ) .

Then the coordinate system

ξ = (w, v, τ, u, z)

⁠, given by formula , admits a center manifold structure around the two points

θ = (w^{*}, w^{*}, 0, v^{*}, τ^{*})

⁠,

(w^{*}, w^{*}, v^{*}, 0, τ^{*})

⁠, and the dynamical system is reduced to that of

(u, z)

⁠. This is confirmed by the argument of the coefficient matrix of the linearization, similar to theorem 2.

4 Numerical Simulations

In the previous section, we showed that the dynamics of $(w, v)$ are fast and those of $(u, z)$ are slow under the coordinate system, equation 3.5. In this section, we verify this fact by numerical simulations.

4.1 Example 1

As the first example, we set the input dimension to be

n = 1

and choose the teacher function

T : R \to R

defined by

T (x) : = 2 tanh (x) - tanh (4 x),

where

tanh

is the hyperbolic tangent function. The shape of

T

is shown in Figure 2 by the solid black line. We set the activation function

ϕ

as

tanh

⁠. Thus, the target function

T

can be represented by the (1-2-1)-perceptron with no bias terms:

f_{(2)} (x; θ) = v_{1} ϕ (w_{1} x) + v_{2} ϕ (w_{2} x),

and the true parameter is

(w_{1}, w_{2}, v_{1}, v_{2}) = (1, 4, 2, - 1)

⁠. We also discard the bias terms

w_{1}^{0}

and

w_{2}^{0}

of the student perceptron. This makes the matrix

H

scalar valued, and hence the assumption of proposition 1 holds trivially.

Figure 2:

View large Download slide

The target function $T (x)$ and the (1-1-1)-perceptron $f_{(1)} (x; θ^{*})$ ⁠, which corresponds to the local minimizer $θ^{*}$ ⁠.

We assume that the data set

{x_{s}}_{s = 1}^{S}

is given and that the probability distribution of

x

is the empirical distribution on that data set. Then the transition formula, equation , of the parameter

θ

is rewritten as

θ^{(t + 1)} = θ^{(t)} - ɛ \frac{1}{S} \sum_{s = 1}^{S} {\frac{\partial ℓ (x_{s}, f_{(2)} (x_{s}; θ))}{\partial θ}|}_{θ = θ^{(t)}} .

In this simulation, we set the size $S$ of the data set to be 1000, and data ${x_{s}}_{s = 1}^{S}$ are drawn identical and independenty distributed (i.i.d.) according to $N (0, 2^{2})$ ⁠. Here, $N (μ, σ^{2})$ denotes the gaussian distribution with mean $μ$ and variance $σ^{2}$ ⁠.

For a data set given as above, we obtained a local minimizer $θ^{*} = (w^{*}, v^{*}) = (0.459, 1.15)$ of $L_{(1)}$ ⁠. The shape of the function that corresponds to the local minimizer is shown by the dashed blue line in Figure 2. The value of $H$ is approximately 0.0472. Since $H > 0$ ⁠, the attractive region is ${θ_{λ} ∣ λ \in (0, 1)}$ ⁠, due to proposition 1.

Figure 3 displays time evolutions of each parameter in the first 1500 iterations from 50 different initial points. We chose an initial parameter

θ^{(0)} = (w_{1}^{(0)}, w_{2}^{(0)}, v_{1}^{(0)}, v_{2}^{(0)})

by

\begin{matrix} w_{1}^{(0)} & = & w^{*} + ζ_{1}, w_{2}^{(0)} = w^{*} + ζ_{2}, \\ v_{1}^{(0)} & = & v^{*} + \frac{1}{2} (ζ_{3} + ζ_{4}), v_{2}^{(0)} = \frac{1}{2} (ζ_{3} - ζ_{4}), \end{matrix}

so that

v = v^{*} + ζ_{3}

and

z = v^{*} + ζ_{4}

⁠, where

ζ_{1}, ζ_{2} \sim U (- 0.2, 0.2)

⁠, and

ζ_{3}, ζ_{4} \sim U (- 0.2, 0.2)

⁠. Here,

U (a, b)

denotes the uniform distribution on the interval

[a, b] \subset R

⁠. We set the learning rate

ɛ

to be 0.05 and the number of iterations to be 20,000. We can see that the parameters

w

and

v

converge to their equilibriums exponentially fast (see Figures 3a and 3b), while

u

and

z

evolve slowly (see Figures 3c and 3d).

Figure 3:

View large Download slide

Time evolutions of each parameter for the first 1500 iterations. Each trajectory of $w$ or $v$ quickly converges to the equilibrium point $w^{*} = 0.459, v^{*} = 1.15$ ⁠, respectively. Trajectories of $u$ and $z$ evolve very slowly compared with $w$ and $v$ ⁠.

Figure 4 shows evolutions on the

(z, | | u | |^{2})

-plane. The red circles in the figure represent initial points. When

w = w^{*}

and

v = v^{*}

⁠, the

z

-axis is a Milnor-like attractor, and the region

| z | < v^{*}

is the attractive part of it. We can check that parameters near the attractive region are trapped and those near the repulsive region are escaping. The intersection point of the line

z = v^{*}

and

z

-axis corresponds to the point

θ = θ_{1}

⁠, the boundary of the attractive and repulsive parts of the Milnor-like attractor. The analytical trajectories, equation , are plotted as dashed blue curves. Numerical evolutions of the parameter follow the analytical trajectories considerably well around

θ = θ_{1}

⁠. We see in the figure that some instances of time evolutions change their direction sharply. This is because the fast dynamics of

w

and

v

are the main dynamics at the beginning of the learning, while the slow dynamics of

u

and

z

become dominant after

w

and

v

converge to the center manifold.

Figure 4:

View large Download slide

Trajectories on the $(z, | | u | |^{2})$ -plane obtained by learning for 20,000 iterations (solid black curves) and analytical trajectories (dashed blue curves) near $θ = θ_{1} = (w^{*}, w^{*}, v^{*}, 0)$ ⁠. Red circles represent initial points.

4.2 Example 2

As the second example, we consider a three-layer perceptron whose input dimension is

n = 2

⁠. Let the teacher function

T : R^{2} \to R

be given by

T (x_{1}, x_{2}) : = 0.75 Sgm (2.5 x_{1} - 2.5 x_{2}) + Sgm (2.5 x_{1} + 2.5 x_{2}) + 0.5,

where

Sgm

is the logistic sigmoidal function, which is defined by

Sgm (x) : = \frac{1}{1 + e^{- 2 x}} = \frac{1}{2} (1 + tanh (x)) .

Figure 5 shows the shape of the teacher function

T (x)

⁠. We use a perceptron with no bias terms also in this simulation and choose

Sgm

as the activation function. Note that a (2-2-1)-perceptron is unable to represent the target function in this case.

Figure 5:

View large Download slide

The target teacher function $T (x)$ ⁠.

Also in this simulation, we assume that the probability distribution of

x

is the empirical distribution on a fixed data set

{x_{s}}_{s = 1}^{S}

⁠. We set the number

S

of the data set to be 1000, and draw a data set

{x_{s}}_{s = 1}^{S}

i.i.d. according to

N (0, I_{2})

⁠, where

I_{2}

denotes the

2 \times 2

identity matrix. We chose a realization

{x_{s}}_{s = 1}^{S}

as above and obtained a local minimizer

θ^{*} = (w^{*}, v^{*})

of

L_{(1)}

⁠, where

w^{*} = (0.399, 0.0652)

and

v^{*} = 2.76

⁠. Figure 6 shows the shape of the (2-1-1)-perceptron corresponding to the local minimizer. The matrix

H

is numerically computed as

(\begin{matrix} - 0.044 & - 0.026 \\ - 0.026 & - 0.20 \end{matrix}) .

Since this

H

is negative definite, the attractive region is

{θ_{λ} ∣ λ \in R ∖ [0, 1]}

due to proposition 1.

Figure 6:

View large Download slide

The (2-1-1)-perceptron $f_{(1)} (x; θ^{*})$ that corresponds to the local minimizer $θ^{*}$ ⁠.

Figures 7a to 7d show time evolutions of each parameter in the first 500 iterations from 50 different initial points. We chose initial parameters of the (2-2-1)-perceptron as

\begin{matrix} w_{1}^{(0)} = w^{*} + ζ_{1}, w_{2}^{(0)} = w^{*} + ζ_{2}, \\ v_{1}^{(0)} = v^{*} + \frac{1}{2} (ζ_{3} + ζ_{4}), v_{2}^{(0)} = \frac{1}{2} (ζ_{3} - ζ_{4}), \end{matrix}

where

ζ_{1}, ζ_{2} \sim U {(- 0.2, 0.2)}^{2}

and

ζ_{3}, ζ_{4} \sim U (- 0.2, 0.2)

⁠. We set the learning rate

ɛ

to be 0.05 and the number of iterations to be 20,000. In this simulation, since

w

and

u

are two-dimensional, their evolutions are not displayed as time series but as trajectories on each plane. The red circles in Figures 7a and 7c represent initial values of

w

and

u

⁠, respectively. Figures 7a and 7b show that the parameters

w

and

v

converge to their equilibrium

w^{*}

and

v^{*}

very quickly. To display how quick the convergence is, the first 30 iterations are dashed in green in Figure 7a. In contrast, Figures 7c and 7d show that the parameters

u

and

z

evolve very slowly.

Figure 7:

View large Download slide

Time evolutions of each parameter for the first 500 iterations. Each trajectory of $w$ or $v$ quickly converges to the equilibrium point $w^{*} = (0.399, 0.0652), v^{*} = 2.76$ respectively. However, trajectories of $u$ and $z$ evolve very slowly compared with $w$ and $v$ ⁠. In panel a, the first 30 iterations are dashed in green to display the speed of convergence, and the red point at the center represents $w = w^{*}$ ⁠. Red circles in panels a and c represent initial points.

Figure 8 plots time evolutions of the parameter

θ

on the

(z, | | u | |^{2})

-plane, which means the plane whose axes indicate the values of

z

and

{∥ u ∥}^{2}

⁠. We can check that parameters near the attractive part

{| z | > v^{*}, | | u | |^{2} = 0}

of the Milnor-like attractor are trapped and that those near the repulsive part

{| z | < v^{*}, | | u | |^{2} = 0}

are escaping. The numerical evolutions follow the analytical flows (dashed blue curves) well also in this case.

Figure 8:

View large Download slide

Trajectories on the $(z, | | u | |^{2})$ -plane obtained by learning for 20,000 iterations (solid black curves) and analytical trajectories near $θ = θ_{1}$ (dashed blue curves). Red circles represent the initial points.

5 Aspects of Online Learning

In this section, we discuss stochastic effects in the learning process. Thus far, we have analyzed the dynamical system, equation , driven by the averaged gradient. In practice, the averaged gradient is estimated by the arithmetic mean of the instantaneous gradient

\partial ℓ (x, f_{(d)} (x; θ)) / \partial θ

over a large number of input data. However, taking the arithmetic mean for each update of the parameter demands high computational cost. In order to reduce the cost, the expectation is often replaced by a single realization of the instantaneous gradient. Such a method is called online learning, a typical stochastic gradient descent method. Mathematically, it is given by

θ^{(t + 1)} = θ^{(t)} - ɛ {\frac{\partial}{\partial θ} ℓ (x_{t}, f_{(d)} (x_{t}; θ))|}_{θ = θ^{(t)}},

(5.1)

where

{x_{t}}_{t}

are i.i.d. realizations of the input data

x

⁠. Unlike the deterministic dynamical system, equation , the system is a randomized dynamical system.

In numerical simulations, we found that sample paths of the online learning seem quite different from trajectories obtained in the averaged gradient descent method. We set the distribution over which the loss function

L_{(1)}

is averaged to be

N (0, 2^{2})

and obtained a local minimizer of

L_{(1)}

as

θ^{*} = (w^{*}, v^{*}) \approx (0.472, 1.13)

⁠. We carried out numerical simulations of the online learning in the same setting as example 1 in section 4. Figure 9a shows numerical trajectories of the averaged gradient descent on the

(z, u)

-plane around

θ = θ_{1}

⁠. In order to approximate the averaged gradient descent sufficiently, we used the empirical distribution on a data set of 10,000 data drawn i.i.d. according to

N (0, 2^{2})

⁠. Figure 9b shows sample paths of the online learning for a common input data sequence

{x_{t}}_{t}

⁠. In contrast to the averaged gradient descent, in the online learning, some sample paths move from region

{| z | > v^{*}}

to

{| z | < v^{*}}

⁠. Such sample paths are observed even when we use another realization of the input data sequence, and its dynamics seems qualitatively different from the averaged one.

Figure 9:

View large Download slide

Trajectories on the $(z, u)$ -plane obtained by the averaged gradient descent method, equation 1.5, and the online learning, equation 5.1, for 20,000 iterations. Red circles represent initial values.

In order to investigate this phenomenon, we observe the evolution of the parameters, again in the coordinate system, equation . Figures 10a to 10d show time evolutions of each parameter in the first 1500 iterations of the online learning. The parameters

(w, v)

evolve very fast compared with

(u, z)

also in this case. However, in this case,

(w, v)

does not converge to its equilibrium point

(w^{*}, v^{*}) \approx (0.472, 1.13)

⁠, but fluctuates stochastically around

(w^{*}, v^{*})

⁠.

Figure 10:

View large Download slide

Time evolutions of each parameter for the first 1500 iterations of the online learning. Each trajectory of $w$ or $v$ fluctuates intensively around the equilibrium point $w^{*} \approx 0.472, v^{*} \approx 1.134$ ⁠, respectively. Trajectories of $u$ and $z$ evolve very slowly compared with $w$ and $v$ in this case.

Based on these observations, we suppose that

w

and

v

run over a sufficiently wide range of their values to be integrated, while

u

and

z

move in a small range. Then we assume that the dynamics of

(u, z)

is integrated with respect to

(w, v)

according to some stationary distribution. We further assume that

(w, v)

are distributed around

(w^{*}, v^{*})

with finite variance. By integrating the Taylor expansions, equations and , with

(w, v)

⁠, we obtain the following dynamical system near

θ = θ_{1}

⁠:

\begin{matrix} \dot{u} & = & \frac{1}{2} (z - v^{*}) H u + C_{1}, \\ \dot{z} & = & \frac{1}{2} u^{T} H u + C_{2} . \end{matrix}

(5.2)

Here,

C_{1}

and

C_{2}

are constants resulting from the variance and covariance of

(w, v)

⁠. Figure 11 shows the analytical trajectories of the dynamical system, equation , where

C_{1} = 1.71 \times 10^{- 4}

and

C_{2} = - 3.06 \times 10^{- 4}

are determined heuristically. One can find that the deterministic dynamical system, equation , gives similar trajectories to sample paths of the online learning presented in Figure 9b.

Figure 11:

Analytical trajectories on the (z,u)-plane given by the dynamical system 5.2 with C1=1.71×10-4 and C2=-3.06×10-4.

View large Download slide

Analytical trajectories on the $(z, u)$ -plane given by the dynamical system 5.2 with $C_{1} = 1.71 \times 10^{- 4}$ and $C_{2} = - 3.06 \times 10^{- 4}$ ⁠.

We deduce that a fluctuation of the parameter around a center manifold causes constants $C_{1}$ and $C_{2}$ working as drift terms and that it makes the dynamics of the online learning qualitatively different from those of the averaged gradient descent. This example suggests that stochastic effects can influence a macroscopic flow of the learning process via a center manifold structure.

6 Conclusion

In this article, we first gave a quick review of a mechanism that causes plateau phenomena in a three-layer perceptron—in particular, how degeneration of hidden units gives rise to a Milnor-like attractor consisting of both attractive and repulsive parts. We next investigated the dynamics of learning around special points on a Milnor-like attractor and proved the existence of the center manifold. We also succeeded in integrating the reduced dynamical system to obtain an analytical form of a trajectory. We performed several numerical simulations to demonstrate the accuracy of our results. As an application of the center manifold structure, we gave an explanation for a characteristic behavior of the online learning.

Unfortunately, the two examples presented in section 4 were the only ones that we could find in which the assumptions of proposition 1 are fulfilled. This might suggest that the appearance of a Milnor-like attractor would be a rather rare situation in a perceptron that has bias terms. In fact, just by replacing the activation function $Sgm$ with $tanh$ in example 2, the matrix $H$ becomes indefinite and the assumption of proposition 1 is violated. Finding more suggestive examples that shed light on the complex behavior of the dynamics of learning is an important subject for future study.

In section 5, we investigated stochastic effects of the online learning from an intermediate viewpoint between fully stochastic and averaged dynamics. We made use of the center manifold of the averaged dynamics and discussed an integration with quickly fluctuating parameters. There have been many reports of qualitative differences between stochastic and deterministic methods; however, there are few general theories for analyzing such dissimilarities. We expect that the intermediate viewpoint in this article can be a clue to clarify stochastic effects in learning.

Appendix: Proofs of Proposition 1 and Theorem 1

This appendix gives proofs of proposition 1 and Theorem 1. The proof of proposition 1 is based on the analysis of the Hessian matrix of $L_{(2)}$ ⁠. However, the Hessian at the point $θ_{λ}$ becomes singular, since $L_{(2)} (θ_{λ})$ is constant along $λ \in R$ ⁠. Thus, we need to take into account higher-order derivatives of $L_{(2)}$ ⁠, which is overlooked in the Fukumizu and Amari (2000). The prototype of theorem 1 was given by Amari et al. (2018); however, they proved it only for a special case. Here, we give a rigorous proof with an additional mild assumption.

Proof of Proposition 1. We introduce a new coordinate system

ξ = (w, v, u, z)

by

\begin{matrix} \{\begin{matrix} w = \frac{v_{1} w_{1} + v_{2} w_{2}}{v_{1} + v_{2}} \\ v = v_{1} + v_{2} \\ u = w_{1} - w_{2} \\ z = v_{1} - v_{2} \end{matrix}, \end{matrix}

(A.1)

where

v_{1} + v_{2} \neq 0

⁠. Under this coordinate system, the point

θ_{λ}

is denoted as

ξ_{λ} = (w^{*}, v^{*}, 0, (2 λ - 1) v^{*})

⁠. Note that each point

ξ = ξ_{λ}

is a critical point of

L_{(2)} (ξ)

by lemma 1. The inverse transformation is given as

\begin{matrix} \{\begin{matrix} w_{1} = w + \frac{v - z}{2 v} u \\ w_{2} = w - \frac{v + z}{2 v} u \\ v_{1} = \frac{v + z}{2} \\ v_{2} = \frac{v - z}{2} \end{matrix} \end{matrix}

We observe the Hessian matrix

Hess (ξ_{λ})

of

L_{(2)} (ξ)

at

ξ_{λ}

for each

λ \in R

⁠. For all

ξ

such that

u = 0

(or, equivalently,

w_{1} = w_{2}

⁠), using the inverse transformation formula above,

\begin{matrix} \frac{\partial L_{(2)}}{\partial u} (ξ) & = & E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ))}{\partial y} \frac{v^{2} - z^{2}}{4 v} (ϕ^{'} (w_{1} \cdot x) - ϕ^{'} (w_{2} \cdot x)) x] \\ = & 0, \\ \frac{\partial L_{(2)}}{\partial z} (ξ) & = & - E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ))}{\partial y} (\frac{v + z}{4 v} ϕ^{'} (w_{1} \cdot x) - \frac{v - z}{4 v} ϕ^{'} (w_{2} \cdot x)) (u \cdot x)] \\ + E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ))}{\partial y} \frac{1}{2} (ϕ (w_{1} \cdot x) - ϕ (w_{2} \cdot x))] \\ = & 0 . \end{matrix}

Here, we left

w_{1}

and

w_{2}

for notational simplicity. We then have

\begin{matrix} \frac{\partial^{2} L_{(2)}}{\partial γ \partial z} (ξ_{λ}) = 0, \\ \frac{\partial^{2} L_{(2)}}{\partial γ \partial u} (ξ_{λ}) = 0, \end{matrix}

where

γ = w, v, z

⁠. Hence, the matrix

Hess (ξ_{λ})

has the form

\begin{matrix} \overset{w, v}{\overset{⏞}{}} \overset{u}{\overset{⏞}{}} \overset{z}{\overset{⏞}{}} \\ \begin{matrix} w, v { \\ u { \\ z { \end{matrix} (\begin{matrix} * & * & 0 & 0 \\ * & 0 & 0 \\ 0 & 0 & * & 0 \\ 0 & 0 & 0 & 0 \end{matrix}) . \end{matrix}

The

(w, v)

-block is equal to the Hessian matrix of

L_{(1)}

at

θ^{*}

and is positive definite. In fact, for any

ξ

such that

u = 0

⁠, noting that

f_{(2)} (x; ξ) \equiv f_{(1)} (x; w, v)

⁠,

\begin{matrix} \frac{\partial L_{(2)}}{\partial w} (ξ) & = & E_{x} [\frac{\partial ℓ (x, f_{(1)} (x; w, v))}{\partial y} v ϕ^{'} (w \cdot x) x] = \frac{\partial L_{(1)}}{\partial w_{(1)}} (w, v), \\ \frac{\partial L_{(2)}}{\partial v} (ξ) & = & E_{x} [\frac{\partial ℓ (x, f_{(1)} (x; w, v))}{\partial y} ϕ (w \cdot x)] = \frac{\partial L_{(1)}}{\partial v_{(1)}} (w, v) . \end{matrix}

The

(w, v)

-block of

Hess (ξ_{λ})

is given by differentiating the equations above with

(w, v)

and thus equal to the Hessian matrix

(\partial^{2} L_{(1)} / \partial θ_{(1)} \partial θ_{(1)}) (θ^{*})

⁠, which is positive definite since

θ_{(1)} = θ^{*}

is a strict local minimizer of

L_{(1)} (θ_{(1)})

⁠.

On the other hand, we have

\begin{matrix} \frac{\partial^{2} L_{(2)}}{\partial u \partial u} (ξ_{λ}) & = & E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ_{λ}))}{\partial y} λ (1 - λ) v^{*} ϕ^{''} (w^{*} \cdot x) x x^{T}] \\ = & λ (1 - λ) H, \end{matrix}

where

H

is the matrix defined in equation . Thus, when

H

is positive definite, the Hessian matrix

Hess (ξ_{λ})

is positive semidefinite if and only if

λ \in [0, 1]

⁠. This proves that

ξ_{λ}

is a saddle point of

L_{(2)} (ξ)

for

λ \in R ∖ [0, 1]

⁠. Similarly, when

H

is negative definite,

ξ_{λ}

is a saddle point of

L_{(2)} (ξ)

for

λ \in (0, 1)

⁠. When

H

is indefinite, so is

Hess (ξ_{λ})

for every

λ \in R ∖ {0, 1}

⁠.

We show that the point

ξ = ξ_{λ}

is a local minimizer of

L_{(2)} (ξ)

for

λ \in (0, 1)

when the matrix

H

is positive definite. For

λ \in (0, 1)

⁠, since the matrix

Hess (ξ_{λ})

is positive semidefinite, the Taylor series of

(L_{(2)} (ξ) - L_{(2)} (ξ_{λ}))

up to the second order is nonnegative in a neighborhood of

ξ = ξ_{λ}

⁠. However, higher-order terms may influence the sign of the Taylor series, since the coefficients of the terms of

z^{2}

and

γ z

are zero for

γ = w, v, u

⁠. Since the terms of

γ^{k} z^{ℓ}

with

k \geq 2

are dominated by the term

γ^{2}

near

ξ = ξ_{λ}

⁠, we check if the coefficients of the terms of

z^{ℓ}

and

γ z^{ℓ}

are equal to zero for all

ℓ \geq 1

⁠. For all

ξ

such that

u = 0

⁠,

f_{(2)} (x; ξ) = v ϕ (w \cdot x)

is a constant as a function of

z

⁠, and so is

L_{(2)} (ξ)

⁠. Hence, we obtain

\begin{matrix} \frac{\partial^{ℓ} L_{(2)}}{\partial z^{ℓ}} (ξ_{λ}) = 0, \\ \frac{\partial^{ℓ + 1} L_{(2)}}{\partial γ \partial z^{ℓ}} (ξ_{λ}) = 0, ℓ \geq 1, \end{matrix}

for

γ = w, v

⁠. Therefore, it suffices to check if

\frac{\partial^{ℓ + 1} L_{(2)}}{\partial u \partial z^{ℓ}} (ξ_{λ}) = 0, ℓ \geq 1 .

Since we have already seen that

(\partial L_{(2)} / \partial u) (ξ) = 0

for all

ξ

such that

u = 0

⁠, this equality is confirmed. Finally, we have shown that the coefficients of higher-order terms are all zero, and that they do not influence the sign of the Taylor series. This shows that

(L_{(2)} (ξ) - L_{(2)} (ξ_{λ}))

is nonnegative near

ξ = ξ_{λ}

⁠, and hence the proof is complete. In the case that the matrix

H

is negative, it is proved similarly.

□

Proof of Theorem 1. Since

v_{1}

and

v_{2}

are no longer scalars, we cannot use the coordinate system, given by equation . Therefore, in order to analyze the Hessian matrix, we introduce another coordinate system

ξ = (w, v, u, z)

as

\begin{matrix} \{\begin{matrix} w = \frac{w_{1} + w_{2}}{2} \\ v = v_{1} + v_{2} \\ u = \frac{w_{1} - w_{2}}{2} \\ z = v_{1} - v_{2} \end{matrix} . \end{matrix}

In this coordinate system, the point

θ_{λ}

is denoted as

ξ_{λ} = (w^{*}, v^{*}, 0, (2 λ - 1) v^{*})

⁠.

Fix $λ \in R$ arbitrarily. We show that the Hessian matrix $Hess (ξ_{λ})$ of $L_{(2)} (ξ)$ at $ξ = ξ_{λ}$ has both positive and negative eigenvalues. It suffices to show that the $(w, v)$ -part of the Hessian is positive definite and that the $(u, z)$ -part is not positive semidefinite. They imply that the full Hessian matrix $Hess (ξ_{λ})$ is neither negative semidefinite nor positive semidefinite, and hence that $Hess (ξ_{λ})$ is indefinite.

One can check that the $(w, v)$ -part of $Hess (ξ_{λ})$ is equal to the Hessian matrix of $L_{(1)}$ at $θ^{*}$ by direct calculation. Since $θ^{*}$ is a strict local minimizer of $L_{(1)}$ ⁠, this is positive definite.

On the other hand, we have

\frac{\partial^{2} L_{(2)}}{\partial z \partial u} (ξ_{λ}) = E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ_{λ}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}],

(A.2)

\frac{\partial^{2} L_{(2)}}{\partial z \partial z} (ξ_{λ}) = O,

(A.3)

and thus the

(u, z)

-part

B

of

Hess (ξ_{λ})

is written as

B = (\begin{array}{c} \frac{\partial^{2} L_{(2)}}{\partial u \partial u} (ξ_{λ}) & E_{x} {[\frac{\partial ℓ (x, f_{(2)} (x; ξ_{λ}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}]}^{T} \\ E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ_{λ}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}] & O \end{array}),

where we treat

\partial ℓ / \partial y

as a column vector. Here, since

f_{(2)} (x; ξ_{λ}) \equiv f_{(1)} (x; θ^{*})

⁠, the nondiagonal block, equation , is equal to the matrix, equation , and thus is nonzero by assumption. Hence, the

(u, z)

-part above is not positive semidifinite. In fact, choosing vectors

a \in R^{m}

and

b \in R^{n + 1}

such that

a^{T} E_{x} [\frac{\partial ℓ (x, f_{(2)} (x; ξ_{λ}))}{\partial y} ϕ^{'} (w^{*} \cdot x) x^{T}] b < 0,

the vector

c_{ɛ} : = {(ɛ b^{T}, a^{T})}^{T}

satisfies

c_{ɛ}^{T} B c_{ɛ} < 0

for sufficiently small

ɛ > 0

⁠. Such vectors

(a, b)

always exist—for instance, by letting

a_{i} = - ρ

⁠,

b_{j} = 1

⁠, and other entries be 0, where

(i, j)

is an index such that

(i, j)

-entry

ρ

of the matrix is nonzero.

□

Acknowledgments

I express my gratitude to Akio Fujiwara for his helpful guidance, discussions, and advice. I thank as well Yuzuru Sato for many discussions and helpful comments. And I thank the anonymous referees for their insightful suggestions to improve this article.

References

Amari

,

S.-I.

,

Ozeki

,

T.

,

Karakida

,

R.

,

Yoshida

,

Y.

, &

Okada

,

M.

(

2018

).

Dynamics of learning in MLP: Natural gradient and singularity revisited

.

Neural Computation

,

30

(

1

),

1

–

33

.

Google Scholar

Carr

,

J.

(

1981

).

Applications of centre manifold theory.

New York

:

Springer

.

Cousseau

,

F.

,

Ozeki

,

T.

, &

Amari

,

S.-I.

(

2008

).

Dynamics of learning in multilayer perceptrons near singularities

.

IEEE Transactions on Neural Networks

,

19

(

8

),

1313

–

1328

.

Google Scholar

Cybenko

,

G.

(

1989

).

Approximation by superpositions of a sigmoidal function

.

Mathematics of Control, Signals, and Systems

,

2

(

4

),

303

–

314

.

Google Scholar

Fukumizu

,

K.

, &

Amari

,

S.-I.

(

2000

).

Local minima and plateaus in hierarchical structures of multilayer perceptrons

.

Neural Networks

,

13

(

3

),

317

–

327

.

Google Scholar

Funahashi

,

K.

(

1989

).

On the approximate realization of continuous mappings by neural networks

.

Neural Networks

,

2

(

3

),

183

–

192

.

Google Scholar

Sonoda

,

S.

, &

Murata

,

N.

(

2017

).

Neural network with unbounded activation functions is universal approximator

.

Applied and Computational Harmonic Analysis

,

43

(

2

),

233

–

268

.

Google Scholar

Wei

,

H.

,

Zhang

,

J.

,

Cousseau

,

F.

,

Ozeki

,

T.

, &

Amari

,

S.-I.

(

2008

).

Dynamics of learning near singularities in layered networks

.

Neural Computation

,

20

(

3

),

813

–

843

.

Google Scholar

2020

Massachusetts Institute of Technology

Center Manifold Analysis of Plateau Phenomena Caused by Degeneration of Three-Layer Perceptron

Abstract

1 Introduction

2 Singular Region and Milnor-Like Attractor

3 Center Manifold of Milnor-Like Attractor

3.1 Brief Review of Center Manifold

3.2 Main Results

3.3 Reduced Dynamical System

3.4 More General Models

4 Numerical Simulations

4.1 Example 1

4.2 Example 2

5 Aspects of Online Learning

6 Conclusion

Appendix: Proofs of Proposition 1 and Theorem 1

Acknowledgments

References

Email alerts

Cited By

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

Center Manifold Analysis of Plateau Phenomena Caused by Degeneration of Three-Layer Perceptron Free

Abstract

1 Introduction

2 Singular Region and Milnor-Like Attractor

3 Center Manifold of Milnor-Like Attractor

3.1 Brief Review of Center Manifold

3.2 Main Results

3.3 Reduced Dynamical System

3.4 More General Models

4 Numerical Simulations

4.1 Example 1

4.2 Example 2

5 Aspects of Online Learning

6 Conclusion

Appendix: Proofs of Proposition 1 and Theorem 1

Acknowledgments

References

Email alerts

Cited By

Related Articles

Related Book Chapters

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

This Feature Is Available To Subscribers Only

Center Manifold Analysis of Plateau Phenomena Caused by Degeneration of Three-Layer Perceptron