On PDE Characterization of Smooth Hierarchical Functions Computed by Neural Networks

Filom, Khashayar; Farhoodi, Roozbeh; Kording, Konrad Paul

doi:10.1162/neco_a_01441

Abstract

Neural networks are versatile tools for computation, having the ability to approximate a broad range of functions. An important problem in the theory of deep neural networks is expressivity; that is, we want to understand the functions that are computable by a given network. We study real, infinitely differentiable (smooth) hierarchical functions implemented by feedforward neural networks via composing simpler functions in two cases: (1) each constituent function of the composition has fewer inputs than the resulting function and (2) constituent functions are in the more specific yet prevalent form of a nonlinear univariate function (e.g., tanh) applied to a linear multivariate function. We establish that in each of these regimes, there exist nontrivial algebraic partial differential equations (PDEs) that are satisfied by the computed functions. These PDEs are purely in terms of the partial derivatives and are dependent only on the topology of the network. Conversely, we conjecture that such PDE constraints, once accompanied by appropriate nonsingularity conditions and perhaps certain inequalities involving partial derivatives, guarantee that the smooth function under consideration can be represented by the network. The conjecture is verified in numerous examples, including the case of tree architectures, which are of neuroscientific interest. Our approach is a step toward formulating an algebraic description of functional spaces associated with specific neural networks, and may provide useful new tools for constructing neural networks.

1 Introduction

1.1 Motivation

A central problem in the theory of deep neural networks is to understand the functions that can be computed by a particular architecture (Raghu, Poole, Kleinberg, Ganguli, & Dickstein, 2017; Poggio, Banburski, & Liao, 2019). Such functions are typically superpositions of simpler functions, that is, compositions of functions of fewer variables. This article aims to study superpositions of real smooth (i.e., infinitely differentiable or

C^{\infty}

⁠) functions that are constructed hierarchically (see Figure 3). Our core thesis is that such functions (also referred to as hierarchical or compositional interchangeably) are constrained in the sense that they satisfy certain partial differential equations (PDEs). These PDEs are dependent only on the topology of the network and could be employed to characterize smooth functions computable by a given network.

Figure 1:

View large Download slide

The architecture on the left (studied in example 1) can compute functions of the form $g (f (x, y), z)$ as in the middle. They involve the smaller class of functions of the form $g (w_{3} f (w_{1} x + w_{2} y + b_{1}) + w_{4} z + b_{2})$ on the right.

Figure 2:

View large Download slide

Implementations of superpositions of the form $F (x, y, z) = g (f (x, y), h (x, z))$ (studied in examples 2 and 7) by three-layer neural networks.

Figure 3:

View large Download slide

The neural network on the left can compute the hierarchical function $F (x_{1}, x_{2}, x_{3}) = f_{1}^{(3)} (f_{1}^{(2)} (f_{1}^{(1)} (x_{1}, x_{2}), f_{2}^{(1)} (x_{2}, x_{3})), f_{2}^{(2)} (f_{2}^{(1)} (x_{2}, x_{3}), f_{3}^{(1)} (x_{3}, x_{1})))$ once appropriate functions are assigned to its nodes as on the right.

1.1.1 Example 1

One of the simplest examples of a superposition is when a trivariate function is obtained from composing two bivariate functions; for instance, let us consider the composition

F (x, y, z) = g (f (x, y), z)

(1.1)

of functions

f = f (x, y)

and

g = g (u, z)

that can be computed by the network in Figure 1. Assuming that all functions appearing here are twice continuously differentiable (or

C^{2}

⁠), the chain rule yields

F_{x} = g_{u} f_{x}, F_{y} = g_{u} f_{y} .

If either

F_{x}

or

F_{y}

– say the former – is nonzero, the equations above imply that the ratio between

F_{x}

and

F_{y}

is independent of

z

⁠:

\frac{F_{y}}{F_{x}} = \frac{f_{y}}{f_{x}} .

(1.2)

Therefore, its derivative with respect to

z

must be identically zero:

{(\frac{F_{y}}{F_{x}})}_{z} = \frac{F_{y z} F_{x} - F_{x z} F_{y}}{{(F_{x})}^{2}} = 0 .

(1.3)

This amounts to

F_{y z} F_{x} = F_{x z} F_{y},

(1.4)

an equation that always holds for functions of form . Notice that one may readily exhibit functions that do not satisfy the necessary PDE constraint

F_{x z} F_{y} = F_{y z} F_{x}

and so cannot be brought into form , for example,¹

x y z + x + y + z .

(1.5)

Conversely, if the constraint

F_{y z} F_{x} = F_{x z} F_{y}

is satisfied and

F_{x}

(or

F_{y}

⁠) is nonzero, we can reverse this processes to obtain a local expression of the form for

F (x, y, z)

⁠. By interpreting the constraint as the independence of

\frac{F_{x}}{F_{y}}

of

z

⁠, one can devise a function

f = f (x, y)

whose ratio of partial derivatives coincides with

\frac{F_{x}}{F_{y}}

(this is a calculus fact; see theorem 5). Now that equation is satisfied, the gradient of

F

may be written as

\nabla F = [\begin{matrix} F_{x} \\ F_{y} \\ F_{z} \end{matrix}] = \frac{F_{x}}{f_{x}} [\begin{matrix} f_{x} \\ f_{y} \\ 0 \end{matrix}] + F_{z} [\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}],

that is, as a linear combination of gradients of

f (x, y)

and

z

⁠. This guarantees that

F (x, y, z)

is (at least locally) a function of the latter two (see the discussion at the beginning of section 3). So there exists a bivariate function

g

defined on a suitable domain with

F (x, y, z) = g (f (x, y), z)

⁠. Later in the article, we generalize this toy example to a characterization of superpositions computed by tree architectures (see theorem 3).

Functions appearing in the context of neural networks are more specific than a general superposition such as equation ; they are predominantly constructed by composing univariate nonlinear activation functions and multivariate linear functions defined by weights and biases. In the case of a trivariate function

F (x, y, z)

⁠, we should replace the representation

g (f (x, y), z)

studied so far with

F (x, y, z) = g (w_{3} f (w_{1} x + w_{2} y + b_{1}) + w_{4} z + b_{2}) .

(1.6)

Assuming that activation functions

f

and

g

are differentiable, now new constraints of the form are imposed. The ratio

\frac{F_{y}}{F_{x}}

is equal to

\frac{w_{2}}{w_{1}}

⁠, hence it is not only independent of

z

as equation suggests, but indeed a constant function. So we arrive at

{(\frac{F_{y}}{F_{x}})}_{x} = {(\frac{F_{y}}{F_{x}})}_{y} = {(\frac{F_{y}}{F_{x}})}_{z} = 0,

or, equivalently,

F_{x y} F_{x} = F_{x x} F_{y}, F_{y y} F_{x} = F_{x y} F_{y}, F_{y z} F_{x} = F_{x z} F_{y} .

Again, these equations characterize differentiable functions of the form ; this is a special case of theorem 7 below.

1.1.2 Example 2

The preceding example dealt with compositions of functions with disjoint sets of variables and this facilitated our calculations. But this is not the case for compositions constructed by most neural networks, for example, networks may be fully connected or may have repeated inputs. For instance, let us consider a superposition of the form

F (x, y, z) = g (f (x, y), h (x, z))

(1.7)

of functions

f (x, y)

⁠,

h (x, z)

⁠, and

g (u, v)

as implemented in Figure 2. Applying the chain rule tends to be more complicated than the case of equation and results in identities

F_{x} = g_{u} f_{x} + g_{v} h_{x}, F_{y} = g_{u} f_{y}, F_{z} = g_{v} h_{z} .

(1.8)

Nevertheless, it is not hard to see that there are again (perhaps cumbersome) nontrivial PDE constraints imposed on the hierarchical function

F

⁠, a fact that will be established generally in theorem 1. To elaborate, notice that identities in equation together imply

F_{x} = A (x, y) F_{y} + B (x, z) F_{z},

(1.9)

where

A : = \frac{f_{x}}{f_{y}}

and

B : = \frac{h_{x}}{h_{z}}

are independent of

z

and

y

⁠, respectively. Repeatedly differentiating this identity (if possible) with respect to

y, z

results in linear dependence relations between partial derivatives of

F

(and hence PDEs) since the number of partial derivatives of

F_{x}

of order at most

n

with respect to

y, z

grows quadratically with

n

⁠, while on the right-hand side, the number of possibilities for coefficients (partial derivatives of

A

and

B

with respect to

y

and

z

⁠, respectively) grows only linearly. Such dependencies could be encoded by the vanishing of determinants of suitable matrices formed by partial derivatives of

F

⁠. In example 7, by pursuing the strategy just mentioned, we complete this treatment of superpositions by deriving the corresponding characteristic PDEs that are necessary and (in a sense) sufficient conditions on

F

that it be in the form of equation . Moreover, in order to be able to differentiate several times, we shall assume that all functions are smooth (or

C^{\infty}

⁠) hereafter.

1.2 Statements of Main Results

Fixing a neural network hierarchy for composing functions, we shall prove that once the constituent functions of corresponding superpositions have fewer inputs (lower arity), there exist universal algebraic partial differential equations (algebraic PDEs) that have these superpositions as their solutions. A conjecture, which we verify in several cases, states that such PDE constraints characterize a generic smooth superposition computable by the network. Here, genericity means a nonvanishing condition imposed on an algebraic expression of partial derivatives. Such a condition has already occurred in example 1 where in the proof of the sufficiency of equation 1.4 for the existence of a representation of the form 1.1 for a function $F (x, y, z)$ ⁠, we assumed either $F_{x}$ or $F_{y}$ is nonzero. Before proceeding with the statements of main results, we formally define some of the terms that have appeared so far.

Terminology

We take all neural networks to be feedforward. A feedforward neural network is an acyclic hierarchical layer to layer scheme of computation. We also include residual networks (ResNets) in this category: an identity function in a layer could be interpreted as a jump in layers. Tree architectures are recurring examples of this kind. We shall always assume that in the first layer, the inputs are labeled by (not necessarily distinct) labels chosen from coordinate functions $x_{1}, \dots, x_{n}$ ⁠, and there is only one node in the output layer. Assigning functions to nodes in layers above the input layer implements a real scalar-valued function $F = F (x_{1}, \dots, x_{n})$ as the superposition of functions appearing at nodes (see Figure 3).
In our setting, an algebraic PDE is a nontrivial polynomial relation such as
$Φ (F_{x_{1}}, \dots, F_{x_{n}}, F_{x_{1}^{2}}, F_{x_{1} x_{2}}, \dots, F_{x^{α}}, \dots) = 0$
(1.10)
among the partial derivatives (up to a certain order) of a smooth function $F = F (x_{1}, \dots, x_{n})$ ⁠. Here, for a tuple $α : = (α_{1}, \dots, α_{n})$ of nonnegative integers, the partial derivative $\frac{\partial^{α_{1} + \dots + α_{n}} F}{\partial x_{1}^{α_{1}} \dots \partial x_{n}^{α_{n}}}$ (which is of order $| α | : = α_{1} + \dots + α_{n}$ ⁠) is denoted by $F_{x^{α}}$ ⁠. For instance, asking for a polynomial expression of partial derivatives of $F$ to be constant amounts to $n$ algebraic PDEs given by setting the first-order partial derivatives of that expression with respect to $x_{1}, \dots, x_{n}$ to be zero.
A nonvanishing condition imposed on smooth functions $F = F (x_{1}, \dots, x_{n})$ is asking for these functions not to satisfy a particular algebraic PDE, namely,
$Ψ (F_{x_{1}}, \dots, F_{x_{n}}, F_{x_{1}^{2}}, F_{x_{1} x_{2}}, \dots, F_{x^{α}}, \dots) \neq 0,$
(1.11)
for a nonconstant polynomial $Ψ$ ⁠. Such a condition could be deemed pointwise since if it holds at a point $p \in R^{n}$ ⁠, it persists throughout a small enough neighborhood. Moreover, equation 1.11 determines an open dense subset of the functional space; so, it is satisfied generically.

Theorem 1.

Let $N$ be a feedforward neural network in which the number of inputs to each node is less than the total number of distinct inputs to the network. Superpositions of smooth functions computed by this network satisfy nontrivial constraints in the form of certain algebraic PDEs that are dependent only on the topology of $N$ ⁠.

In the context of deep learning, the functions applied at each node are in the form of

y \mapsto σ (〈 w, y 〉);

(1.12)

that is, they are obtained by applying an activation function

σ

to a linear functional

y \mapsto 〈 w, y 〉

⁠. Here, as usual, the bias term is absorbed into the weight vector. The bias term could also be excluded via composing

σ

with a translation since throughout our discussion, the only requirement for a function

σ

to be the activation function of a node is smoothness, and activation functions are allowed to vary from a node to another. In our setting,

σ

in equation could be a polynomial or a sigmoidal function such as hyperbolic tangent or logistic functions, but not ReLU or maxout activation functions. We shall study functions computable by neural networks as either superpositions of arbitrary smooth functions or as superpositions of functions of the form , which is a more limited regime. Indeed, the question of how well arbitrary compositional functions, which are the subject of theorem 1, may be approximated by a deep network has been studied in the literature (Mhaskar, Liao, & Poggio, 2017; Poggio, Mhaskar, Rosasco, Miranda, & Liao, 2017).

In order to guarantee the existence of PDE constraints for superpositions, theorem 1 assumes a condition on the topology of the network. However, theorem 2 states that by restricting the functions that can appear in the superposition, one can still obtain PDE constraints even for a fully connected multilayer perceptron:

Theorem 2.

Let $N$ be an arbitrary feedforward neural network with at least two distinct inputs, with smooth functions of the form 1.12 applied at its nodes. Any function computed by this network satisfies nontrivial constraints in the form of certain algebraic PDEs that are dependent only on the topology of $N$ ⁠.

1.2.1 Example 3

As the simplest example of PDE constraints imposed on compositions of functions of the form , recall that d'Alembert's solution to the wave equation,

u_{t t} = c^{2} u_{x x},

(1.13)

is famously given by superpositions of the form

f (x + c t) + g (x - c t)

⁠. This function can be implemented by a network with two inputs

x, t

and with one hidden layer in which the activation functions

f, g

are applied (see Figure 4). Since we wish for a PDE that works for this architecture universally, we should get rid of

c

⁠. The PDE may be written as

\frac{u_{t t}}{u_{x x}} = c^{2}

⁠; that is the ratio

\frac{u_{t t}}{u_{x x}}

must be constant. Hence, for our purposes, the wave equation should be written as

{(\frac{u_{t t}}{u_{x x}})}_{x} = {(\frac{u_{t t}}{u_{x x}})}_{t} = 0

⁠, or equivalently,

u_{x t t} u_{x x} - u_{t t} u_{x x x} = 0, u_{t t t} u_{x x} - u_{t t} u_{x x t} = 0 .

A crucial point to notice is that the constant

c^{2}

is nonnegative; thus an inequality of the form

\frac{u_{x x}}{u_{t t}} \geq 0

or

u_{x x} u_{t t} \geq 0

is imposed as well. In example 11, we visit this network again and study functions of the form

F (x, t) = σ (a^{''} f (a x + b t) + b^{''} g (a^{'} x + b^{'} t))

(1.14)

via a number of equalities and inequalities involving partial derivatives of

F

⁠.

Figure 4:

View large Download slide

The neural network on the left can compute the function $F (x, t) = σ (a^{''} f (a x + b t) + b^{''} g (a^{'} x + b^{'} t))$ once, as on the right, the activation functions $σ, f, g$ and appropriate weights are assigned to the nodes. Such functions are the subject of examples 3 and 11.

The preceding example suggests that smooth functions implemented by a neural network may be required to obey a nontrivial algebraic partial differential inequality (algebraic PDI). So it is convenient to have the following setup of terminology.

Terminology

An algebraic PDI is an inequality of the form
$Θ (F_{x_{1}}, \dots, F_{x_{n}}, F_{x_{1}^{2}}, F_{x_{1} x_{2}}, \dots, F_{x^{α}}, \dots) > 0$
(1.15)
involving partial derivatives (up to a certain order) where $Θ$ is a real polynomial.

Remark 1.

Without any loss of generality, we assume that the PDIs are strict since a nonstrict one such as $Θ \geq 0$ could be written as the union of $Θ > 0$ and the algebraic PDE $Θ = 0$ ⁠.

Theorem 1 and example 1 deal with superpositions of arbitrary smooth functions while theorem 2 and example 3 are concerned with superpositions of a specific class of smooth functions, functions of the form 1.12. In view of the necessary PDE constraints in both situations, the following question then arises: Are there sufficient conditions in the form of algebraic PDEs and PDIs that guarantee a smooth function can be represented, at least locally, by the neural network in question?

Conjecture 1.

Let $N$ be a feedforward neural network whose inputs are labeled by the coordinate functions $x_{1}, \dots, x_{n}$ ⁠. Suppose we are working in the setting of one of theorems 1 or 2. Then there exist

finitely many nonvanishing conditions ${\{Ψ_{i} ({(F_{x^{α}})}_{| α | \leq r}) \neq 0\}}_{i}$
finitely many algebraic PDEs ${\{Φ_{j} ({(F_{x^{α}})}_{| α | \leq r}) = 0\}}_{j}$
finitely many algebraic PDIs ${\{Θ_{k} ({(F_{x^{α}})}_{| α | \leq r}) > 0\}}_{k}$

with the following property: For any arbitrary point $p \in R^{n}$ ⁠, the space of smooth functions $F = F (x_{1}, \dots, x_{n})$ defined in a vicinity^¹ of $p$ that satisfy $Ψ_{i} \neq 0$ at $p$ and are computable by $N$ (in the sense of the regime under consideration) is nonvacuous and is characterized by PDEs $Φ_{j} = 0$ and PDIs $Θ_{k} > 0$ ⁠.

To motivate the conjecture, notice that it claims the existence of functionals

{\{F \mapsto Ψ_{i} ({(F_{x^{α}})}_{| α | \leq r})\}}_{i}, {\{F \mapsto Φ_{j} ({(F_{x^{α}})}_{| α | \leq r})\}}_{j}, {\{F \mapsto Θ_{k} ({(F_{x^{α}})}_{| α | \leq r})\}}_{k},

which are polynomial expressions of partial derivatives, and hence continuous in the

C^{r}

-norm,^² such that in the space of functions computable by

N

⁠, the open dense^³ subset given by

{\{Ψ_{i} \neq 0\}}_{i}

can be described in terms of finitely many equations and inequalities as the locally closed subset

{\{Φ_{j} = 0\}}_{j} ⋃ {\{Θ_{k} > 0\}}_{k} .

(Also see corollary 1.) The usage of

C^{r}

-norm here is novel. For instance, with respect to

L^{p}

-norms, the space of functions computable by

N

lacks such a description and often has undesirable properties like nonclosedness (Petersen, Raslan, & Voigtlaender, 2020). Besides, describing the functional space associated with a neural network

N

with finitely many equations and inequalities also has an algebraic motivation: it is reminiscent of the notion of a semialgebraic set from real algebraic geometry. To elaborate, take the activation functions to be polynomials. Such neural networks have been studied in the literature (Du & Lee, 2018; Soltanolkotabi, Javanmard, & Lee, 2018; Venturi, Bandeira, & Bruna, 2018; Kileel, Trager, & Bruna, 2019). By bounding the degrees of constituent functions of superpositions computed by a polynomial neural network, the functional space formed by these superpositions sits inside a finite-dimensional ambient space of real polynomials and is hence finite-dimensional and amenable to techniques of algebraic geometry. One can, for instance, in each degree associate a functional variety to a neural network

N

whose dimension could be interpreted as a measure of expressive power (Kileel et al., 2019). Our approach to describing real functions computable by neural networks via PDEs and PDIs has ramifications to the study of polynomial neural networks as well. Indeed, if

F = F (x_{1}, \dots, x_{n})

is a polynomial, an algebraic PDE of the form translates to a polynomial equation of the coefficients of

F

⁠, and the condition that an algebraic PDI such as equation is valid throughout

R^{n}

can again be described via equations and inequalities involving the coefficients of

F

(see examples 12 and 13). A notable feature here is the claim of the existence of a universal characterization dependent only on the architecture from which a description as a semialgebraic set could be read off in any degree.

Conjecture 1 is settled in (Farhoodi, Filom, Jones, and Körding, 2019) for trees (a particular type of architectures) with distinct inputs, a situation in which no PDI is required, and the inequalities should be taken to be trivial. Throughout the article, the conjecture above will be established for a number of architectures; in particular, we shall characterize tree functions (cf. theorems 3 and 4 below).

1.3 Related Work

There is an extensive literature on the expressive power of neural networks. Although shallow networks with sigmoidal activation functions can approximate any continuous function on compact sets (Cybenko, 1989; Hornik, Stinchcombe, & White, 1989; Hornik, 1991; Mhaskar, 1996), this cannot be achieved without the hidden layer getting exponentially large (Eldan & Shamir, 2016; Telgarsky, 2016; Mhaskar et al., 2017; Poggio et al., 2017). Many articles thus try to demonstrate how the expressive power is affected by depth. This line of research draws on a number of different scientific fields including algebraic topology (Bianchini & Scarselli, 2014), algebraic geometry (Kileel et al., 2019), dynamical systems (Chatziafratis, Nagarajan, Panageas, & Wang, 2019), tensor analysis (Cohen, Sharir, & Shashua, 2016), Vapnik–Chervonenkis theory (Bartlett, Maiorov, & Meir, 1999), and statistical physics (Lin, Tegmark, & Rolnick, 2017). One approach is to argue that deeper networks are able to approximate or represent functions of higher complexity after defining a “complexity measure” (Bianchini & Scarselli, 2014; Montufar, Pascanu, Cho, & Bengio, 2014; Poole, Lahiri, Raghu, Sohl-Dickstein, & Ganguli, 2016; Telgarsky, 2016; Raghu et al., 2017). Another approach more in line with this article is to use the “size” of an associated functional space as a measure of representation power. This point of view is adapted in Farhoodi et al. (2019) by enumerating Boolean functions, and in Kileel et al. (2019) by regarding dimensions of functional varieties as such a measure.

A central result in the mathematical study of superpositions of functions is the celebrated Kolmogorov-Arnold representation theorem (Kolmogorov, 1957), which resolves (in the context of continuous functions) the thirteenth problem on Hilbert's famous list of 23 major mathematical problems (Hilbert, 1902). The theorem states that every continuous function

F (x_{1}, \dots, x_{n})

on the closed unit cube may be written as

F (x_{1}, \dots, x_{n}) = \sum_{i = 1}^{2 n + 1} f_{i} (\sum_{j = 1}^{n} ϕ_{i, j} (x_{j}))

(1.16)

for suitable continuous univariate functions

f_{i}

⁠,

ϕ_{i, j}

defined on the unit interval. (See Vituškin and Henkin, 1967, chap. 1, or Vituškin, 2004, for a historical account.) In more refined versions of this theorem (Sprecher, 1965; Lorentz, 1966), the outer functions

f_{i}

are arranged to be the same, and the inner ones

ϕ_{i, j}

are be taken to be in the form of

λ_{j} ϕ_{i}

with

λ_{j}

's and

ϕ_{i}

's independent of

F

⁠. Based on the existence of such an improved representation, Hecht-Nielsen argued that any continuous function

F

can be implemented by a three-layer neural network whose weights and activation functions are determined by the representation (Hecht-Nielsen, 1987). On the other hand, it is well known that even when

F

is smooth, one cannot arrange for functions appearing in representation to be smooth (Vituškin, 1964). As a matter of fact, there exist continuously differentiable functions of three variables that cannot be represented as sums of superpositions of the form

g (f (x, y), z)

with

f

and

g

being continuously differentiable as well (Vituškin, 1954) whereas in the continuous category, one can write any trivariate continuous functions as a sum of nine superpositions of the form

g (f (x, y), z)

(Arnold, 2009b). Due to this emergence of nondifferentiable functions, it has been argued that Kolmogorov-Arnold's theorem is not useful for obtaining exact representations of functions via networks (Girosi & Poggio, 1989), although it may be used for approximation (Kůrková, 1991, 1992). More on algorithmic aspects of the theorem and its applications to the network theory can be found in Brattka (2007).

Focusing on a superposition

\begin{matrix} F & = & f_{1}^{(L)} (f_{1}^{(L - 1)} (f_{a_{1}}^{(L - 2)} (\dots), \dots), \dots, f_{j}^{(L - 1)} (f_{a_{j}}^{(L - 2)} (\dots), \dots), \dots, \\ f_{N_{L - 1}}^{(L - 1)} (f_{a_{N_{L - 1}}}^{(L - 2)} (\dots), \dots)) \end{matrix}

(1.17)

of smooth functions (which can be computed by a neural network as in Figure 3), the chain rule provides descriptions for partial derivatives of

F

in terms of partial derivatives of functions

f_{j}^{(i)}

that constitute the superposition. The key insight behind the proof of theorem 1 is that when the former functions have fewer variables compared to

F

⁠, one may eliminate the derivatives of

f_{j}^{(i)}

's to obtain relations among partial derivatives of

F

⁠. This idea of elimination has been utilized in Buck (1981b) and Rubel (1981) to prove the existence of universal algebraic differential equations whose

C^{\infty}

solutions are dense in the space of continuous functions. The fact that there will be constraints imposed on derivatives of a function

F

that is written as a superposition of differentiable functions was employed by Hilbert himself to argue that certain analytic functions of three variables are not superpositions of analytic functions of two variables (Arnold, 2009a, p. 28), and by Ostrowski to exhibit an analytic bivariate function that cannot be represented as a superposition of univariate smooth functions and multivariate algebraic functions due to the fact that it does not satisfy any nontrivial algebraic PDE (Vituškin, 2004, p. 14; Ostrowski, 1920). The novelty of our approach is to adapt this point of view to demonstrate theoretical limitations of smooth functions that neural networks compute either as a superposition as in theorem 1 or as compositions of functions of the form as in theorem 2, and to try to characterize these functions via calculating PDE constraints that are sufficient too (cf. conjecture 1). Furthermore, necessary PDE constraints enable us to easily exhibit functions that cannot be computed by a particular architecture; see example 1. This is reminiscent of the famous Minsky XOR Theorem (Minsky & Papert, 2017). An interesting nonexample from the literature is

F (x, y, z) = x y + y z + z x

which cannot be written as a superposition of the form even in the continuous category (Pólya & Szegö, 1945; Buck, 1979, 1981a; von Golitschek, 1980; Arnold, 2009a).

To the best of our knowledge, the closest mentions of a characterization of a class of superpositions by necessary and sufficient PDE constraints in the literature are papers (Buck, 1979, 1981a) by R. C. Buck. The first one (along with its earlier version, Buck, 1976) characterizes superpositions of the form $g (f (x, y), z)$ in a similar fashion as example 1. Also in those papers, superpositions such as $g (f (x, y), h (x, z))$ (which appeared in example 2) are discussed although only the existence of necessary PDE constraints is shown; see (Buck, 1979, lemma 7), and (Buck, 1981a, p. 141). We exhibit a PDE characterization for superpositions of this form in example 7. These papers also characterize sufficiently differentiable nomographic functions of the form $σ (f (x) + g (y))$ and $σ (f (x) + g (y) + h (z))$ ⁠.

A special class of neural network architectures is provided by rooted trees where any output of a layer is passed to exactly one node from one of the layers above (see Figure 8). Investigating functions computable by trees is of neuroscientific interest because the morphology of the dendrites of a neuron processes information through a tree that is often binary (Kollins & Davenport, 2005; Gillette & Ascoli, 2015). Assuming that the inputs to a tree are distinct, in our previous work (Farhoodi et al., 2019), we have completely characterized the corresponding superpositions through formulating necessary and sufficient PDE constraints; a result that answers conjecture 1 in positive for such architectures.

Figure 5:

View large Download slide

Theorems 3 and 4 impose constraints 1.18 and 1.19 for any three leaves $x_{i}$ ⁠, $x_{j}$ ⁠, and $x_{k}$ ⁠. In the former theorem, the constraint should hold whenever (as on the left) there exists a rooted full subtree separating $x_{i}$ and $x_{j}$ from $x_{k}$ ⁠, while in the latter theorem, the constraint is imposed for certain other triples as well (as on the right).

Figure 6:

View large Download slide

Theorem 4 imposes constraint 1.20 for any four leaves $x_{i}, x_{i^{'}}$ and $x_{j}, x_{j^{'}}$ that belong to two different rooted full subtrees emanating from a node.

Remark 2.

The characterization suggested by the theorem below is a generalization of example 1 which was concerned with smooth superpositions of the form 1.1. The characterization of such superpositions as solutions of PDE 1.4 has also appeared in a paper (Buck, 1979) that we were not aware of while writing (Farhoodi et al., 2019).

Figure 7:

View large Download slide

A multilayer neural network can be expanded to a tree. The figure is adapted from Farhoodi et al. (2019).

Figure 8:

View large Download slide

A tree architecture and the relevant terminology.

Theorem 3

(Farhoodi et al., 2019). Let

T

be a rooted tree with

n

leaves that are labeled by the coordinate functions

x_{1}, \dots, x_{n}

⁠. Let

F = F (x_{1}, \dots, x_{n})

be a smooth function implemented on this tree. Then for any three leaves of

T

corresponding to variables

x_{i}, x_{j}, x_{k}

of

F

with the property that there is a (rooted full) subtree of

T

containing the leaves

x_{i}, x_{j}

while missing the leaf

x_{k}

(see Figure 5),

F

must satisfy

F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}} .

(1.18)

Conversely, a smooth function

F

defined in a neighborhood of a point

p \in R^{n}

can be implemented by the tree

T

provided that equation holds for any triple

(x_{i}, x_{j}, x_{k})

of its variables with the above property; and moreover, the non-vanishing conditions below are satisfied:

For any leaf $x_{i}$ with siblings either $F_{x_{i}} (p) \neq 0$ or there is a sibling leaf $x_{i^{'}}$ with $F_{x_{i^{'}}} (p) \neq 0$ ⁠.

This theorem was formulated in Farhoodi et al. (2019) for binary trees and in the context of analytic functions (and also that of Boolean functions). Nevertheless, the proof carries over to the more general setting above. Below, we formulate the analogous characterization of functions that trees compute via composing functions of the form 1.12. Proofs of theorems 3 and 4 are presented in section 4.

Theorem 4.

Let $T$ be a rooted tree admitting $n$ leaves that are labeled by the coordinate functions $x_{1}, \dots, x_{n}$ ⁠. We formulate the following constraints on smooth functions $F = F (x_{1}, \dots, x_{n})$ ⁠:

For any two leaves $x_{i}$ and $x_{j}$ of $T$ ⁠, we have
$F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}$
(1.19)
for any other leaf $x_{k}$ of $T$ that is not a leaf of a (rooted full) subtree that has exactly one of $x_{i}$ or $x_{j}$ (see Figure 5). In particular, equation 1.19 holds for any $x_{k}$ if the leaves $x_{i}$ and $x_{j}$ are siblings, and for any $x_{i}$ and $x_{j}$ if the leaf $x_{k}$ is adjacent to the root of $T$ ⁠.
For any two (rooted full) subtrees $T_{1}$ and $T_{2}$ that emanate from a node of $T$ (see Figure 6), we have
$\begin{matrix} F_{x_{i}} F_{x_{j}} [F_{x_{i} x_{i^{'}} x_{j^{'}}} F_{x_{j}} + F_{x_{i} x_{i^{'}}} F_{x_{j} x_{j^{'}}} - F_{x_{i} x_{j^{'}}} F_{x_{j} x_{i^{'}}} - F_{x_{i}} F_{x_{j} x_{i^{'}} x_{j^{'}}}] \\ = (F_{x_{i} x_{i^{'}}} F_{x_{j}} - F_{x_{i}} F_{x_{j} x_{i^{'}}}) (F_{x_{i} x_{j^{'}}} F_{x_{j}} + F_{x_{i}} F_{x_{j} x_{j^{'}}}) \end{matrix}$
(1.20)
if $x_{i}$ ⁠, $x_{i^{'}}$ are leaves of $T_{1}$ and $x_{j}$ ⁠, $x_{j^{'}}$ are leaves of $T_{2}$ ⁠.

These constraints are satisfied if $F (x_{1}, \dots, x_{n})$ is a superposition of functions of the form $y \mapsto σ (〈 w, y 〉)$ according to the hierarchy provided by $T$ ⁠. Conversely, a smooth function $F$ defined on an open box-like region^⁴ $B \subseteq R^{n}$ can be written as such a superposition on $B$ provided that the constraints 1.19 and 1.20 formulated above hold and, moreover, the nonvanishing conditions below are satisfied throughout $B$ ⁠:

For any leaf $x_{i}$ with siblings either $F_{x_{i}} \neq 0$ or there is a sibling leaf $x_{i^{'}}$ with $F_{x_{i^{'}}} \neq 0$ ⁠;
For any leaf $x_{i}$ without siblings $F_{x_{i}} \neq 0$ ⁠.

The constraints that appeared in theorems 3 and 4 may seem tedious, but they can be rewritten more conveniently once the intuition behind them is explained. Assuming that partial derivatives do not vanish (a nonvanishing condition) so that division is allowed, equations and may be written as

{(\frac{F_{x_{i}}}{F_{x_{j}}})}_{x_{k}} = 0 \Leftrightarrow {(\frac{F_{x_{i}}}{F_{x_{k}}})}_{x_{j}} = {(\frac{F_{x_{j}}}{F_{x_{k}}})}_{x_{i}},

(1.21)

while equation is

{(\frac{{(\frac{F_{x_{i}}}{F_{x_{j}}})}_{x_{i^{'}}}}{\frac{F_{x_{i}}}{F_{x_{j}}}})}_{x_{j^{'}}} = 0 .

(1.22)

Equation simply states that the ratio

\frac{F_{x_{i}}}{F_{x_{j}}}

is independent of

x_{k}

⁠. Notice that in comparison with theorem 3, theorem 7, requires the equation

F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}

to hold in a greater generality and for more triples

(x_{i}, x_{j}, x_{k})

of leaves (see Figure 5).^⁵ The second simplified equation , holds once the function

\frac{F_{x_{i}}}{F_{x_{j}}}

of

(x_{1}, \dots, x_{n})

may be split into a product such as

q_{1} (\dots, x_{i}, \dots, x_{i^{'}}, \dots) q_{2} (\dots, x_{j}, \dots, x_{j^{'}}, \dots) .

Lemma 4 discusses the necessity and sufficiency of these equations for the existence of such a splitting.

Remark 3.

A significant feature of theorem 7 is that once the appropriate conditions are satisfied on a box-like domain, the smooth function under consideration may be written as a superposition of the desired form on the entirety of that domain. On the contrary, theorem 3 is local in nature.

Aside from neuroscientific interest, studying tree architectures is important also because any neural network can be expanded into a tree network with repeated inputs through a procedure called TENN (the Tree Expansion of the Neural Network; see Figure 7). Tree architectures with repeated inputs are relevant in the context of neuroscience too because the inputs to neurons may be repeated (Schneider-Mizell et al., 2016; Gerhard, Andrade, Fetter, Cardona, & Schneider-Mizell, 2017). We have already seen an example of a network along with its TENN in Figure 2. Both networks implement functions of the form $F (x, y, z) = g (f (x, y), h (x, z))$ ⁠. Even for this simplest example of a tree architecture with repeated inputs, the derivation of characteristic PDEs is computationally involved and will be done in example 7. This verifies conjecture 1 for the tree that appeared in Figure 2.

1.4 Outline of the Article

Theorems 1 and 2 are proven in section 2 where it is established that in each setting, there are necessary PDE conditions for expressibility of smooth functions by a neural network. In section 3 we verify conjecture 1 in several examples by characterizing computable functions via PDE constraints that are necessary and (given certain nonvanishing conditions) sufficient. This starts by studying tree architectures in section 3.1. In example 7, we finish our treatment of a tree function with repeated inputs initiated in example 2; and, moreover, we present a number of examples to exhibit the key ideas of the proofs of theorems 3 and 4, which are concerned with tree functions with distinct inputs. The section then proceeds with switching from trees to other neural networks in section 3.2 where, building on example 3, example 11 demonstrates why the characterization claimed by conjecture 1 involves inequalities. We end section 3 with a brief subsection on PDE constraints for polynomial neural networks. Examples in section 3.1 are generalized in the next section to a number of results establishing conjecture 1 for certain families of tree architectures: Proofs of theorems 3 and 4 are presented in section 4. The last section is devoted to few concluding remarks. There are two appendices discussing technical proofs of propositions and lemmas (appendix A), and the basic mathematical background on differential forms (appendix B).

2 Existence of PDE Constraints

The goal of the section is to prove theorems 1 and 2. Lemma 1 below is our main tool for establishing the existence of constraints:

Lemma 1.

Any collection

p_{1} (t_{1}, \dots, t_{m}), \dots, p_{l} (t_{1}, \dots, t_{m})

of polynomials on

m

indeterminates are algebraically dependent provided that

l > m

⁠. In other words, if

l > m

⁠, there exists a nonconstant polynomial

Φ = Φ (s_{1}, \dots, s_{l})

dependent only on the coefficients of

p_{i}

's for which

Φ (p_{1} (t_{1}, \dots, t_{m}), \dots, p_{l} (t_{1}, \dots, t_{m})) \equiv 0 .

Proof.

For a positive integer $a$ ⁠, there are precisely $(\binom{a + l}{l})$ monomials such as $p_{1}^{a_{1}} \dots p_{l}^{a_{l}}$ with their total degree $a_{1} + \dots + a_{l}$ not greater than $a$ ⁠. But each of them is a polynomial of $t_{1}, \dots, t_{m}$ of total degree at most $a d$ where $d : = max {deg p_{1}, \dots, deg p_{l}}$ ⁠. For $a$ large enough, $(\binom{a + l}{l})$ is greater than $(\binom{a d + m}{m})$ because the degree of the former as a polynomial of $a$ is $l$ ⁠, while the degree of the latter is $m$ ⁠. For such an $a$ ⁠, the number of monomials $p_{1}^{a_{1}}, \dots, p_{l}^{a_{l}}$ is larger than the dimension of the space of polynomials of $t_{1}, \dots, t_{m}$ of total degree at most $a d$ ⁠. Therefore, there exists a linear dependency among these monomials that amounts to a nontrivial polynomial relation among $p_{1}, \dots, p_{l}$ ⁠.

Proof of Theorem 1.

Let

F = F (x_{1}, \dots, x_{n})

be a superposition of smooth functions

f_{1}^{(1)}, \dots, f_{N_{1}}^{(1)}; \dots; f_{1}^{(i)}, \dots, f_{N_{i}}^{(i)}; \dots; f_{1}^{(L)}

(2.1)

according to the hierarchy provided by

N

where

f_{1}^{(i)}, \dots, f_{N_{i}}^{(i)}

are the functions appearing at the neurons of the

i

th layer above the input layer (in the last layer,

f_{N_{L} = 1}^{(L)}

appears at the output neuron). The total number of these functions is

N : = N_{1} + \dots + N_{L}

⁠, namely, the number of the neurons of the network. By the chain rule, any partial derivative

F_{x^{α}}

of the superposition may be described as a polynomial of partial derivatives of order not greater than

| α |

of functions that appeared in equation . These polynomials are determined solely by how neurons in consecutive layers are connected to each other, that is, the architecture. The function

F

of

n

variables admits

(\binom{r + n}{n}) - 1

partial derivatives (excluding the function itself) of order at most

r

⁠, whereas the same number for any of the functions listed in equation is at most

(\binom{r + n - 1}{n - 1}) - 1

because by the hypothesis, each of them is dependent on less than

n

variables. Denote the partial derivatives of order at most

r

of functions

f_{j}^{(i)}

(evaluated at appropriate points as required by the chain rule) by indeterminates

t_{1}, \dots, t_{m}

⁠. Following the previous discussion, one has

m \leq N ((\binom{r + n - 1}{n - 1}) - 1)

⁠. Hence, the chain rule describes the partial derivatives of order not greater than

r

of

F

as polynomials (dependent only on the architecture of

N

⁠) of

t_{1}, \dots, t_{m}

⁠. Invoking lemma 1, the partial derivatives of

F

are algebraically dependent once

(\binom{r + n}{n}) - 1 > N ((\binom{r + n - 1}{n - 1}) - 1) .

(2.2)

Indeed, the inequality holds for

r

large enough since the left-hand side is a polynomial of degree

n

of

r

⁠, while the similar degree for the right-hand side is

n - 1

⁠.

Proof of Theorem 2.

In this case

F = F (x_{1}, \dots, x_{n})

is a superposition of functions of the form

\begin{matrix} σ_{1}^{(1)} (〈 w_{1}^{(1)}, . 〉), \dots, σ_{N_{1}}^{(1)} (〈 w_{N_{1}}^{(1)}, . 〉); \dots; σ_{1}^{(i)} (〈 w_{1}^{(i)}, . 〉), \dots, σ_{N_{i}}^{(i)} (〈 w_{N_{i}}^{(i)}, . 〉); \\ \dots; σ_{1}^{(L)} (〈 w_{1}^{(L)}, . 〉) \end{matrix}

(2.3)

appearing at neurons. The

j

th neuron of the

i

th layer above the input layer (⁠

1 \leq i \leq N, 1 \leq j \leq N_{i}

⁠) corresponds to the function

σ_{j}^{(i)} (〈 w_{j}^{(i)}, . 〉)

where a univariate smooth activation function

σ_{j}^{(i)}

is applied to the inner product of the weight vector

w_{j}^{(i)}

with the vector formed by the outputs of neurons in the previous layer which are connected to the neuron of the

i

th layer. We proceed as in the proof of theorem 1. The chain rule describes each partial derivative

F_{x^{α}}

as a polynomial, dependent only on the architecture, of components of vectors

w_{j}^{(i)}

along with derivatives of functions

σ_{j}^{(i)}

up to order at most

| α |

(each evaluated at an appropriate point). The total number of components of all weight vectors coincides with the total number of connections (edges of the underlying graph), and the number of the derivatives of activation functions is the number of neurons times

| α |

⁠. We denote the total number of connections and neurons by

C

and

N

⁠, respectively. There are

(\binom{r + n}{n}) - 1

partial derivatives

F_{x^{α}}

of order at most

r

(i.e.,

| α | \leq r

⁠) of

F

and, by the previous discussion, each of them may be written as a polynomial of

C + N r

quantities given by components of weight vectors and derivatives of activation functions. Lemma 1 implies that these partial derivatives of

F

are algebraically dependent provided that

(\binom{r + n}{n}) - 1 > N r + C,

(2.4)

an inequality that holds for sufficiently large

r

as the degree of the left-hand side with respect to

r

is

n > 1

⁠.

Corollary 1.

Let $N$ be a feedforward neural network whose inputs are labeled by the coordinate functions $x_{1}, \dots, x_{n}$ and satisfies the hypothesis of either of theorems 1 or 2. Define the positive integer $r$ as

$r = n (# neurons - 1)$ in the case of theorem 1
$r = max (⌊ n {(# neurons)}^{\frac{1}{n - 1}} ⌋, # connections) + 2$ in the case of theorem 2,

where $# connections$ and $# neurons$ are, respectively, the number of edges of the underlying graph of $N$ and the number of its vertices above the input layer. Then the smooth functions $F = F (x_{1}, \dots, x_{n})$ computable by $N$ satisfy nontrivial algebraic partial differential equations of order $r$ ⁠. In particular, the subspace formed by these functions lies in a subset of positive codimension, which is closed with respect to the $C^{r}$ -norm.

Proof.

One only needs to verify that for the values of

r

provided by the corollary the inequalities and are valid. The former holds if

\frac{(\binom{r + n}{n})}{(\binom{r + n - 1}{n - 1})} = \frac{r + n}{n}

is not smaller than

N

⁠, that is, if

r \geq n (N - 1)

⁠. As for equation , notice that

(\binom{r + n}{n}) - 1 - N r \geq \frac{r^{n}}{n!} - N r = r (\frac{r^{n - 1}}{n!} - N);

hence, it suffices to have

r (\frac{r^{n - 1}}{n!} - N) > C

⁠. This holds if

r > C

and

\frac{r^{n - 1}}{n!} - N \geq 1

⁠. The latter inequality is valid once

r \geq n . N^{\frac{1}{n - 1}} + 2

⁠, since then:

\begin{matrix} \frac{r^{n - 1}}{n!} & = & {(\frac{r}{{(n!)}^{\frac{1}{n - 1}}})}^{n - 1} \geq {(\frac{r}{n})}^{n - 1} \geq {(N^{\frac{1}{n - 1}} + \frac{2}{n})}^{n - 1} \\ \geq & N + \frac{2 (n - 1)}{n} . N^{\frac{n - 2}{n - 1}} \geq N + 1 . \end{matrix}

Remark 4.

It indeed follows from the arguments above that there is a multitude of algebraically independent PDE constraints. By a simple dimension count, this number is $((\binom{r + n}{n}) - 1) - N ((\binom{r + n - 1}{n - 1}) - 1)$ in the first case of corollary 1 and $((\binom{r + n}{n}) - 1) - N r$ in the second case.

Remark 5.

The approach here merely establishes the existence of nontrivial algebraic PDEs satisfied by the superpositions. These are not the simplest PDEs of this kind and hence are not the best candidates for the purpose of characterizing superpositions. For instance, for superpositions 1.7, which networks in Figure 2 implement, one has $n = 3$ and $# neurons = 3$ ⁠. Corollary 1 thus guarantees that these superpositions satisfy a sixth-order PDE. But in example 7, we shall characterize them via two fourth-order PDEs (compare with Buck, 1979, lemma 7).

Remark 6.

Prevalent smooth activation functions such as the logistic function

\frac{1}{1 + e^{- x}}

or tangent hyperbolic

\frac{e^{x} - e^{- x}}{e^{x} + e^{- x}}

satisfy certain autonomous algebraic ODEs. Corollary 1 could be improved in such a setting. If each activation function

σ = σ (x)

appearing in equation satisfies a differential equation of the form

\frac{d^{k} σ}{d x^{k}} = p (σ, \frac{d σ}{d x}, \dots, \frac{d^{k - 1} σ}{d x^{k - 1}})

where

p

is a polynomial, one can change equation to

(\binom{r + n}{n}) - 1 > N k_{max} + C

where

k_{max}

is the maximum order of ODEs that activation functions in equation satisfy.

3 Toy Examples

This section examines several elementary examples demonstrating how one can derive a set of necessary or sufficient PDE constraints for an architecture. The desired PDEs should be universal, that is, purely in terms of the derivatives of the function $F$ that is to be implemented and not dependent on any weight vector, activation function, or a function of lower dimensionality that has appeared at a node. In this process, it is often necessary to express a smooth function in terms of other functions. If $k < n$ and $f (x_{1}, \dots, x_{n})$ is written as $g (ξ_{1}, \dots, ξ_{k})$ throughout an open neighborhood of a point $p \in R^{n}$ where each $ξ_{i} = ξ_{i} (x_{1}, \dots, x_{n})$ is a smooth function, the gradient of $f$ must be a linear combination of those of $ξ_{1}, \dots, ξ_{k}$ due to the chain rule. Conversely, if $\nabla f \in Span {\nabla ξ_{1}, \dots, \nabla ξ_{k}}$ near $p$ ⁠, by the inverse function theorem, one can extend $(ξ_{1}, \dots, ξ_{k})$ to a coordinate system $(ξ_{1}, \dots, ξ_{k}; ξ_{k + 1}, \dots, ξ_{n})$ on a small enough neighborhood of $p$ provided that $\nabla ξ_{1} (p), \dots, \nabla ξ_{k} (p)$ are linearly independent; a coordinate system in which the partial derivative $f_{ξ_{i}}$ vanishes for $k < i \leq n$ ⁠; the fact that implies $f$ can be expressed in terms of $ξ_{1}, \dots, ξ_{k}$ near $p$ ⁠. Subtle mathematical issues arise if one wants to write $f$ as $g (ξ_{1}, \dots, ξ_{k})$ on a larger domain containing $p$ ⁠:

A $k$ -tuple $(ξ_{1}, \dots, ξ_{k})$ of smooth functions defined on an open subset $U$ of $R^{n}$ whose gradient vector fields are linearly independent at all points cannot necessarily be extended to a coordinate system $(ξ_{1}, \dots, ξ_{k}; ξ_{k + 1}, \dots, ξ_{n})$ for the whole $U$ ⁠. As an example, consider $r = \sqrt{x^{2} + y^{2}}$ whose gradient is nonzero at any point of $R^{2} - {(0, 0)}$ ⁠, but there is no smooth function $h : R^{2} - {(0, 0)} \to R$ with $\nabla h \neg ∥ \nabla r$ throughout $R^{2} - {(0, 0)}$ ⁠. The level set $r = 1$ is compact, and so the restriction of $h$ to it achieves its absolute extrema, and at such points $\nabla h = λ \nabla f$ (⁠ $λ$ is the Lagrange multiplier).
Even if one has a coordinate system $(ξ_{1}, \dots, ξ_{k}; ξ_{k + 1}, \dots, ξ_{n})$ on a connected open subset $U$ of $R^{n}$ ⁠, a smooth function $f : U \to R$ with $f_{ξ_{k + 1}}, \dots, f_{ξ_{n}} \equiv 0$ cannot necessarily be written globally as $f = g (ξ_{1}, \dots, ξ_{k})$ ⁠. One example is the function
$f (x, y) : = \{\begin{matrix} 0 & if x \leq 0 \\ e^{\frac{- 1}{x}} & if x > 0, y > 0 \\ - e^{\frac{- 1}{x}} & if x > 0, y < 0 \end{matrix}$
defined on the open subset $R^{2} - [0, \infty) \subset R^{2}$ for which $f_{y} \equiv 0$ ⁠. It may only locally be written as $f (x, y) = g (x)$ ⁠; there is no function $g : R \to R$ with $f (x, y) = g (x)$ for all $(x, y) \in R^{2} - [0, \infty)$ ⁠. Defining $g (x_{0})$ as the value of $f$ on the intersection of its domain with the vertical line $x = x_{0}$ does not work because, due to the shape of the domain, such intersections may be disconnected. Finally, notice that $f$ ⁠, although smooth, is not analytic (⁠ $C^{ω}$ ⁠); indeed, examples of this kind do not exist in the analytic category.

This difficulty of needing a representation $f = g (ξ_{1}, \dots, ξ_{k})$ that remains valid not just near a point but over a larger domain comes up only in the proof of theorem 4 (see remark 3); the representations we work with in the rest of this section are all local. The assumption about the shape of the domain and the special form of functions 1.12 allows us to circumvent the difficulties just mentioned in the proof of theorem 4. Below we have two related lemmas that we use later.

Lemma 2.

Let $B$ and $T$ be a box-like region in $R^{n}$ and a rooted tree with the coordinate functions $x_{1}, \dots, x_{n}$ labeling its leaves as in theorem 7. Suppose a smooth function $F = F (x_{1}, \dots, x_{n})$ on $B$ is implemented on $T$ via assigning activation functions and weights to the nodes of $T$ ⁠. If $F$ satisfies the nonvanishing conditions described at the end of theorem 7, then the level sets of $F$ are connected and $F$ can be extended to a coordinate system $(F, F_{2}, \dots, F_{n})$ for $B$ ⁠.

Lemma 3.

A smooth function $F (x_{1}, \dots, x_{n})$ of the form $σ (a_{1} x_{1} + \dots + a_{n} x_{n})$ satisfies $F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}$ for any $1 \leq i, j, k \leq n$ ⁠. Conversely, if $F$ has a first-order partial derivative $F_{x_{j}}$ which is nonzero throughout an open box-like region $B$ in its domain, each identity $F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}$ could be written as ${(\frac{F_{x_{i}}}{F_{x_{j}}})}_{x_{k}} = 0$ ⁠; that is, for any $1 \leq i \leq n$ ⁠, the ratio $\frac{F_{x_{i}}}{F_{x_{j}}}$ should be constant on $B$ ⁠, and such requirements guarantee that $F$ admits a representation of the form $σ (a_{1} x_{1} + \dots + a_{n} x_{n})$ on $B$ ⁠.

In view of the discussion so far, it is important to know when a smooth vector field,

V (x_{1}, \dots, x_{n}) = {[V_{1} (x_{1}, \dots, x_{n}) \dots V_{n} (x_{1}, \dots, x_{n})]}^{T},

(3.1)

on an open subset

U \subset R^{n}

is locally given by a gradient. Clearly, a necessary condition is to have

{(V_{i})}_{x_{j}} = {(V_{j})}_{x_{i}} \forall i, j \in {1, \dots, n} .

(3.2)

It is well known that if

U

is simply connected, this condition is sufficient too and guarantees the existence of a smooth potential function

ξ

on

U

satisfying

\nabla ξ = V

(Pugh, 2002). A succinct way of writing equation is

d ω = 0

where

ω

is defined as the differential form:

ω : = V_{1} d x_{1} + \dots + V_{n} d x_{n} .

(3.3)

Here is a more subtle question also pertinent to our discussion: When may

V

be rescaled to a gradient vector field? As the reader may recall from the elementary theory of differential equations, for a planer vector field, such a rescaling amounts to finding an integration factor for the corresponding first order ODE (Boyce & DiPrima, 2012). It turns out that the answer could again be encoded in terms of differential forms:

Theorem 5.

A smooth vector field $V$ is parallel to a gradient vector field near each point only if the corresponding differential 1-form $ω$ satisfies $ω \land d ω = 0$ ⁠. Conversely, if $V$ is nonzero at a point $p \in R^{n}$ in the vicinity of which $ω \land d ω = 0$ holds, there exists a smooth function $ξ$ defined on a suitable open neighborhood of $p$ that satisfies $V ∥ \nabla ξ \neq 0$ ⁠. In particular, in dimension 2, a nowhere vanishing vector field $V$ is locally parallel to a nowhere vanishing gradient vector field, while in dimension 3, that is the case if and only if $V . curl V = 0$ ⁠.

A proof and background on differential forms are provided in appendix B.

3.1 Trees with Four Inputs

We begin with officially defining the terms related to tree architectures (see Figure 8).

Terminology

A tree is a connected acyclic graph. Singling out a vertex as its root turns it into a directed acyclic graph in which each vertex has a unique predecessor/parent. We take all trees to be rooted. The following notions come up frequently:
- Leaf: a vertex with no successor/child.
- Node: a vertex that is not a leaf, that is, has children.
- Sibling leaves: leaves with the same parent.
- Subtree: all descendants of a vertex along with the vertex itself. Hence in our convention, all subtrees are full and rooted.
To implement a function, the leaves pass the inputs to the functions assigned to the nodes. The final output is received from the root.

The first example of the section elucidates theorem 3.

3.1.1 Example 4

Let us characterize superpositions

F (x, y, z, w) = g (f (x, y), z, w)

of smooth functions

f, g

⁠, which correspond to the first tree architecture in Figure 9. Necessary PDE constraints are more convenient to write for certain ratios. So to derive them, we assume for a moment that first-order partial derivatives of

F

are nonzero, although by a simple continuity argument, the constraints will hold regardless. Computing the numerator and the denominator of

\frac{F_{x}}{F_{y}}

via the chain rule indicates that this ratio coincides with

\frac{f_{x}}{f_{y}}

and is, hence, independent of

z, w

⁠. One thus obtains

{(\frac{F_{y}}{F_{x}})}_{z} = 0, {(\frac{F_{y}}{F_{x}})}_{w} = 0,

or, equivalently,

F_{y z} F_{x} = F_{x z} F_{y}, F_{y w} F_{x} = F_{x w} F_{y} .

Assuming

F_{x} \neq 0

⁠, the preceding constraints are sufficient. The gradient

\nabla F

is parallel with

[\begin{matrix} 1 \\ \frac{F_{y}}{F_{x}} \\ \frac{F_{z}}{F_{x}} \\ \frac{F_{w}}{F_{x}} \end{matrix}]

where the second entry

\frac{F_{y}}{F_{x}}

is dependent only on

x

and

y

and thus may be written as

\frac{F_{y}}{F_{x}} = \frac{f_{y}}{f_{x}}

for an appropriate bivariate function

f = f (x, y)

defined throughout a small enough neighborhood of the point under consideration (at which

F_{x}

is assumed to be nonzero). Such a function exists due to theorem 5. Now we have

\nabla F ∥ [\begin{matrix} 1 \\ \frac{f_{y}}{f_{x}} \\ 0 \\ 0 \end{matrix}] + \frac{F_{z}}{F_{x}} [\begin{matrix} 0 \\ 0 \\ 1 \\ 0 \end{matrix}] + \frac{F_{w}}{F_{x}} [\begin{matrix} 0 \\ 0 \\ 0 \\ 1 \end{matrix}] \in Span \{\nabla f, \nabla z, \nabla w\},

which guarantees that

F (x, y, z, w)

may be written as a function of

f (x, y), z, w

⁠.

Figure 9:

View large Download slide

Two tree architectures with four distinct inputs. Examples 4, 5, and 6 characterize functions computable by them.

The next two examples serve as an invitation to the proof of theorem 4 in section 4 and are concerned with trees illustrated in Figure 9.

3.1.2 Example 5

Let us study the example above in the regime of activation functions. The goal is to characterize functions of the form

F (x, y, z, w) = σ (τ (a x + b y) + c z + d w)

⁠. The ratios

\frac{F_{y}}{F_{x}}, \frac{F_{z}}{F_{w}}

must be constant while

\frac{F_{x}}{F_{z}}

and

\frac{F_{x}}{F_{w}}

are dependent merely on

x, y

as they are equal to

\frac{a}{c} τ^{'} (a x + b y)

and

\frac{a}{d} τ^{'} (a x + b y)

⁠, respectively. Equating the corresponding partial derivatives with zero, we obtain the following PDEs:

\begin{matrix} F_{x y} F_{x} = F_{x x} F_{y}, F_{y y} F_{x} = F_{x y} F_{y}, F_{y z} F_{x} = F_{x z} F_{y}, F_{y w} F_{x} = F_{x w} F_{y}; \\ F_{x z} F_{w} = F_{x w} F_{z}, F_{y z} F_{w} = F_{y w} F_{z}, F_{z z} F_{w} = F_{z w} F_{z}, F_{z w} F_{w} = F_{w w} F_{z}; \\ F_{x z} F_{z} = F_{z z} F_{x}, F_{x w} F_{z} = F_{z w} F_{x}; F_{x z} F_{w} = F_{z w} F_{x}, F_{x w} F_{w} = F_{w w} F_{x} . \end{matrix}

One can easily verify that they always hold for functions of the form above. We claim that under the assumptions of

F_{x} \neq 0

and

F_{w} \neq 0

⁠, these conditions guarantee the existence of a local representation of the form

σ (τ (a x + b y) + c z + d w)

of

F

⁠. Denoting

\frac{F_{x}}{F_{w}}

by

β (x, y)

and the constant functions

\frac{F_{y}}{F_{x}}

and

\frac{F_{z}}{F_{w}}

by

c_{1}

and

c_{2}

⁠, respectively, we have

\nabla F = [\begin{matrix} F_{x} \\ F_{y} \\ F_{z} \\ F_{w} \end{matrix}] ∥ [\begin{matrix} \frac{F_{x}}{F_{w}} \\ \frac{F_{y}}{F_{w}} \\ \frac{F_{z}}{F_{w}} \\ 1 \end{matrix}] = [\begin{matrix} β (x, y) \\ c_{1} β (x, y) \\ c_{2} \\ 1 \end{matrix}] ∥ \nabla (f (x, y) + c_{2} z + w),

where

\nabla f = [\begin{matrix} β (x, y) \\ c_{1} β (x, y) \end{matrix}]

⁠. Such a potential function

f

for

[\begin{matrix} β (x, y) \\ c_{1} β (x, y) \end{matrix}] = [\begin{matrix} \frac{F_{x}}{F_{w}} \\ \frac{F_{y}}{F_{w}} \end{matrix}]

exists since

{(\frac{F_{x}}{F_{w}})}_{y} = {(\frac{F_{y}}{F_{w}})}_{x} \Leftrightarrow {(\frac{F_{y}}{F_{x}})}_{w} = 0,

and it must be in the form of

τ (a x + b y)

as

\frac{f_{y}}{f_{x}} = c_{1}

is constant (see lemma 3). Thus,

F

is a function of

τ (a x + b y) + c_{2} z + w

because the gradients are parallel.

The next example is concerned with the symmetric tree in Figure 9. We shall need the following lemma:

Lemma 4.

Suppose a smooth function

q = q (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

is written as a product

q_{1} (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}) q_{2} (y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

(3.4)

of smooth functions

q_{1}, q_{2}

⁠. Then

q q_{y_{a}^{(1)} y_{b}^{(2)}} = q_{y_{a}^{(1)}} q_{y_{b}^{(2)}}

for any

1 \leq a \leq n_{1}

and

1 \leq b \leq n_{2}

⁠. Conversely, for a smooth function

q

defined on an open box-like region

B_{1} \times B_{2} \subseteq R^{n_{1}} \times R^{n_{2}}

⁠, once

q

is nonzero, these identities guarantee the existence of such a product representation on

B_{1} \times B_{2}

⁠.

3.1.3 Example 6

We aim for characterizing smooth functions of four variables of the form

F (x, y, z, w) = σ (τ_{1} (a x + b y) + τ_{2} (c z + d w))

⁠. Assuming for a moment that all first-order partial derivatives are nonzero, the ratios

\frac{F_{y}}{F_{x}}, \frac{F_{z}}{F_{w}}

must be constant while

\frac{F_{x}}{F_{w}}

is equal to

\frac{a τ_{1}^{'} (a x + b y)}{d τ_{2}^{'} (c z + d w)}

and hence (along with its constant multiples

\frac{F_{x}}{F_{z}}, \frac{F_{y}}{F_{z}}, \frac{F_{y}}{F_{w}}

⁠) splits into a product of bivariate functions of

x, y

and

z, w

⁠, a requirement that by lemma 4 is equivalent to the following identities:

\begin{matrix} \frac{F_{x}}{F_{w}} {(\frac{F_{x}}{F_{w}})}_{x z} = {(\frac{F_{x}}{F_{w}})}_{x} {(\frac{F_{x}}{F_{w}})}_{z}, \frac{F_{x}}{F_{w}} {(\frac{F_{x}}{F_{w}})}_{x w} = {(\frac{F_{x}}{F_{w}})}_{x} {(\frac{F_{x}}{F_{w}})}_{w}, \\ \frac{F_{x}}{F_{w}} {(\frac{F_{x}}{F_{w}})}_{y z} = {(\frac{F_{x}}{F_{w}})}_{y} {(\frac{F_{x}}{F_{w}})}_{z}, \frac{F_{x}}{F_{w}} {(\frac{F_{x}}{F_{w}})}_{y w} = {(\frac{F_{x}}{F_{w}})}_{y} {(\frac{F_{x}}{F_{w}})}_{w} . \end{matrix}

After expanding and cross-multiplying, the identities above result in PDEs of the form imposed on

F

that hold for any smooth function of the form

F (x, y, z, w) = σ (τ_{1} (a x + b y) + τ_{2} (c z + d w))

⁠. Conversely, we claim that if

F_{x} \neq 0

and

F_{w} \neq 0

⁠, then the constraints we have noted guarantee that

F

locally admits a representation of this form. Denoting the constants

\frac{F_{y}}{F_{x}}

and

\frac{F_{z}}{F_{w}}

by

c_{1}

and

c_{2}

⁠, respectively, and writing

\frac{F_{x}}{F_{w}} \neq 0

in the split form

\frac{β (x, y)}{γ (z, w)}

⁠, we obtain

\nabla F = [\begin{matrix} F_{x} \\ F_{y} \\ F_{z} \\ F_{w} \end{matrix}] ∥ [\begin{matrix} \frac{F_{x}}{F_{w}} \\ \frac{F_{y}}{F_{w}} \\ \frac{F_{z}}{F_{w}} \\ 1 \end{matrix}] = [\begin{matrix} \frac{β (x, y)}{γ (z, w)} \\ c_{1} \frac{β (x, y)}{γ (z, w)} \\ c_{2} \\ 1 \end{matrix}] ∥ [\begin{matrix} β (x, y) \\ c_{1} β (x, y) \\ c_{2} γ (z, w) \\ γ (z, w) \end{matrix}] .

We desire functions

f = f (x, y)

and

g = g (z, w)

with

\nabla f = [\begin{matrix} β (x, y) \\ c_{1} β (x, y) \end{matrix}]

and

\nabla g = [\begin{matrix} c_{2} γ (z, w) \\ γ (z, w) \end{matrix}]

⁠, because then,

\nabla F ∥ \nabla (f (x, y) + g (z, w))

and hence

F = σ (f (x, y) + g (z, w))

for an appropriate

σ

⁠. Notice that

f (x, y)

and

g (z, w)

are automatically in the forms of

τ_{1} (a x + b y)

and

τ_{2} (c z + d w)

because

\frac{f_{y}}{f_{x}} = c_{1}

and

\frac{f_{z}}{f_{w}} = c_{2}

are constants (see lemma 3). To establish the existence of

f

and

g

⁠, one should verify the integrability conditions

β_{y} = c_{1} β_{x}

and

c_{2} γ_{w} = γ_{z}

⁠. We only verify the first one; the second one is similar. Notice that

\frac{F_{y}}{F_{x}} = c_{1}

is constant, and

\frac{F_{x}}{F_{w}} = \frac{β (x, y)}{γ (z, w)}

implies that

β_{x} = β \frac{{(\frac{F_{x}}{F_{w}})}_{x}}{\frac{F_{x}}{F_{w}}}

while

β_{y} = β \frac{{(\frac{F_{x}}{F_{w}})}_{y}}{\frac{F_{x}}{F_{w}}}

⁠. So the question is whether

\frac{F_{y}}{F_{x}} {(\frac{F_{x}}{F_{w}})}_{x} = {(\frac{F_{y}}{F_{x}} \frac{F_{x}}{F_{w}})}_{x} = {(\frac{F_{y}}{F_{w}})}_{x}

and

{(\frac{F_{x}}{F_{w}})}_{y}

coincide, which is the case since

{(\frac{F_{y}}{F_{w}})}_{x} = {(\frac{F_{x}}{F_{w}})}_{y}

can be rewritten as

{(\frac{F_{y}}{F_{x}})}_{w} = 0

⁠.

Remark 7.

Examples 5 and 6 demonstrate an interesting phenomenon: one can deduce nontrivial facts about the weights once a formula for the implemented function is available. In example 5, for a function $F (x, y, z, w) = σ (τ (a x + b y) + c z + d w)$ ⁠, we have $\frac{F_{y}}{F_{x}} \equiv \frac{b}{a}$ and $\frac{F_{z}}{F_{w}} \equiv \frac{c}{d}$ ⁠. The same identities are valid for functions of the form $F (x, y, z, w) = σ (τ_{1} (a x + b y) + τ_{2} (c z + d w))$ in example 6.^⁶ This seems to be a direction worthy of study. In fact, there are papers discussing how a neural network may be “reverse-engineered” in the sense that the architecture of the network is determined from the knowledge of its outputs, or the weights and biases are recovered without the ordinary training process involving gradient descent algorithms (Fefferman & Markel, 1994; Dehmamy, Rohani, & Katsaggelos, 2019; Rolnick & Kording, 2019). In our approach, the weights appearing in a composition of functions of the form $y \mapsto σ (〈 w, y 〉)$ could be described (up to scaling) in terms of partial derivatives of the resulting superposition.

3.1.4 Example 7

Let us go back to example 2. In (Farhoodi et al., 2019, c, 7.2), a PDE constraint on functions of the form is obtained via differentiating equation several times and forming a matrix equation, which implies that a certain determinant of partial derivatives must vanish. The paper then raises the question of existence of PDE constraints that are both necessary and sufficient. The goal of this example is to derive such a characterization. Applying differentiation operators

\partial_{y}

⁠,

\partial_{z}

⁠, and

\partial_{y z}

to equation results in

[\begin{matrix} F_{y} & F_{z} & 0 & 0 \\ F_{y y} & F_{y z} & F_{y} & 0 \\ F_{y z} & F_{z z} & 0 & F_{z} \\ F_{y y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}] [\begin{matrix} A \\ B \\ A_{y} \\ B_{z} \end{matrix}] = [\begin{matrix} F_{x} \\ F_{x y} \\ F_{x z} \\ F_{x y z} \end{matrix}] .

If this matrix is nonsingular, a nonvanishing condition, Cramer's rule provides descriptions of

A, B

in terms of partial derivatives of

F

⁠, and then

A_{z} = B_{y} = 0

yield PDE constraints. Reversing this procedure, we show that these conditions are sufficient too. Let us assume that

\begin{matrix} Ψ & : = & |\begin{matrix} F_{y} & F_{z} & 0 & 0 \\ F_{y y} & F_{y z} & F_{y} & 0 \\ F_{y z} & F_{z z} & 0 & F_{z} \\ F_{y y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}| \\ = & {(F_{y})}^{2} F_{z} F_{y z z} - {(F_{y})}^{2} F_{y z} F_{z z} - F_{y} {(F_{z})}^{2} F_{y y z} + {(F_{z})}^{2} F_{y z} F_{y y} \neq 0 . \end{matrix}

(3.5)

Notice that this condition is nonvacuous for functions

F (x, y, z)

of the form since they include all functions of the form

g (y, z)

⁠. Then the linear system

[\begin{matrix} F_{x} \\ F_{x y} \\ F_{x z} \\ F_{x y z} \end{matrix}] = [\begin{matrix} F_{y} & F_{z} & 0 & 0 \\ F_{y y} & F_{y z} & F_{y} & 0 \\ F_{y z} & F_{z z} & 0 & F_{z} \\ F_{y y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}] [\begin{matrix} A \\ B \\ C \\ D \end{matrix}]

(3.6)

may be solved as

\begin{matrix} A & = & \frac{|\begin{matrix} F_{x} & F_{z} & 0 & 0 \\ F_{x y} & F_{y z} & F_{y} & 0 \\ F_{x z} & F_{z z} & 0 & F_{z} \\ F_{x y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}|}{|\begin{matrix} F_{y} & F_{z} & 0 & 0 \\ F_{y y} & F_{y z} & F_{y} & 0 \\ F_{y z} & F_{z z} & 0 & F_{z} \\ F_{y y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}|} \\ = & \frac{1}{Ψ} [- F_{y} {(F_{z})}^{2} F_{x y z} + F_{y} F_{z} F_{x z} F_{y z} + F_{x} F_{y} F_{z} F_{y z z} \\ - F_{x} F_{y} F_{y z} F_{z z} + {(F_{z})}^{2} F_{x y} F_{y z} - F_{x} F_{z} {(F_{y z})}^{2}] \end{matrix}

(3.7)

and

\begin{matrix} B & = & \frac{|\begin{matrix} F_{y} & F_{x} & 0 & 0 \\ F_{y y} & F_{x y} & F_{y} & 0 \\ F_{y z} & F_{x z} & 0 & F_{z} \\ F_{y y z} & F_{x y z} & F_{y z} & F_{y z} \end{matrix}|}{|\begin{matrix} F_{y} & F_{z} & 0 & 0 \\ F_{y y} & F_{y z} & F_{y} & 0 \\ F_{y z} & F_{z z} & 0 & F_{z} \\ F_{y y z} & F_{y z z} & F_{y z} & F_{y z} \end{matrix}|} \\ = & \frac{1}{Ψ} [{(F_{y})}^{2} F_{z} F_{x y z} - {(F_{y})}^{2} F_{x z} F_{y z} - F_{y} F_{z} F_{x y} F_{y z} \\ - F_{x} F_{y} F_{z} F_{y y z} + F_{x} F_{y} {(F_{y z})}^{2} + F_{x} F_{z} F_{y y} F_{y z}] . \end{matrix}

(3.8)

Denote the numerators of and by

Ψ_{1}

and

Ψ_{2}

⁠, respectively:

\begin{matrix} Ψ_{1} & = & - F_{y} {(F_{z})}^{2} F_{x y z} + F_{y} F_{z} F_{x z} F_{y z} + F_{x} F_{y} F_{z} F_{y z z} \\ - F_{x} F_{y} F_{y z} F_{z z} + {(F_{z})}^{2} F_{x y} F_{y z} - F_{x} F_{z} {(F_{y z})}^{2}, \\ Ψ_{2} & = & {(F_{y})}^{2} F_{z} F_{x y z} - {(F_{y})}^{2} F_{x z} F_{y z} - F_{y} F_{z} F_{x y} F_{y z} \\ - F_{x} F_{y} F_{z} F_{y y z} + F_{x} F_{y} {(F_{y z})}^{2} + F_{x} F_{z} F_{y y} F_{y z} . \end{matrix}

(3.9)

Requiring

A = \frac{Ψ_{1}}{Ψ}

and

B = \frac{Ψ_{2}}{Ψ}

to be independent of

z

and

y

⁠, respectively, amounts to

Φ_{1} : = {(Ψ_{1})}_{z} Ψ - Ψ_{1} Ψ_{z} = 0, Φ_{2} : = {(Ψ_{2})}_{y} Ψ - Ψ_{2} Ψ_{y} = 0 .

(3.10)

A simple continuity argument demonstrates that the constraints

Φ_{1} = 0

and

Φ_{2} = 0

above are necessary even if the determinant vanishes: if

Ψ

is identically zero on a neighborhood of a point

p \in R^{3}

⁠, the identities obviously hold throughout that neighborhood. Another possibility is that

Ψ (p) = 0

⁠, but there is a sequence

{p_{n}}_{n}

of nearby points with

p_{n} \to p

and

Ψ (p_{n}) \neq 0

⁠. Then the polynomial expressions

Φ_{1}

⁠,

Φ_{2}

of partial derivatives vanish at any

p_{n}

and hence at

p

by continuity.

To finish the verification of conjecture 1 for superpositions of the form , one should establish that PDEs

Φ_{1} = 0

⁠,

Φ_{2} = 0

from equation are sufficient for the existence of such a representation provided that the nonvanishing condition

Ψ \neq 0

from equation holds. In that case, the functions

A

and

B

from equations and satisfy equation . According to theorem 5, there exist smooth locally defined

f (x, y)

and

h (x, z)

with

\frac{f_{x}}{f_{y}} = A (x, y)

and

\frac{h_{x}}{h_{z}} = B (x, z)

⁠. We have:

\begin{matrix} \nabla F & = & [\begin{matrix} A (x, y) F_{y} + B (x, z) F_{z} \\ F_{y} \\ F_{z} \end{matrix}] = F_{y} [\begin{matrix} A (x, y) \\ 1 \\ 0 \end{matrix}] + F_{z} [\begin{matrix} B (x, z) \\ 0 \\ 1 \end{matrix}] \\ = & \frac{F_{y}}{f_{y}} [\begin{matrix} f_{x} \\ f_{y} \\ 0 \end{matrix}] + \frac{F_{z}}{h_{z}} [\begin{matrix} h_{x} \\ 0 \\ h_{z} \end{matrix}] \in Span {\nabla f, \nabla h}; \end{matrix}

hence,

F

can be written as a function

g (f (x, y), h (x, z))

of

f

and

h

for an appropriate

g

⁠.

3.1.5 Example 8

We now turn to the asymmetric tree with four repeated inputs in Figure 10 with the corresponding superpositions,

F (x, y, z) = g (x, f (h (x, y), z)) .

(3.11)

In our treatment here, the steps are reversible, and we hence derive PDE constraints that are simultaneously necessary and sufficient. The existence of a representation of the form for

F (x, y, z)

is equivalent to the existence of a locally defined coordinate system,

(ξ : = x, ζ, η),

with respect to which

F_{η} = 0

⁠; moreover,

ζ = ζ (x, y, z)

must be in the form of

f (h (x, y), z)

⁠, which, according to example 1, is the case if and only if

{(\frac{ζ_{y}}{ζ_{x}})}_{z} = 0

⁠. Here, we assume that

ζ_{x}, ζ_{y} \neq 0

so that

\frac{ζ_{y}}{ζ_{x}}

is well defined and

\nabla ξ, \nabla ζ

are linearly independent. We denote the preceding ratio by

β = β (x, y) \neq 0

⁠. Conversely, theorem 5 guarantees that there exists

ζ

with

\frac{ζ_{y}}{ζ_{x}} = β

for any smooth

β (x, y)

⁠. The function

F

could be locally written as a function of

ξ = x

and

ζ

if and only if

\nabla F = [\begin{matrix} F_{x} \\ F_{y} \\ F_{z} \end{matrix}] \in Span \{\nabla x, \nabla ζ = [\begin{matrix} ζ_{x} \\ ζ_{y} = β (x, y) ζ_{x} \\ ζ_{z} \end{matrix}]\} .

Clearly, this occurs if and only if

\frac{F_{z}}{F_{y}}

coincides with

\frac{ζ_{z}}{ζ_{y}}

⁠. Therefore, one only needs to arrange for

β (x, y)

so that the vector field

\frac{1}{ζ_{x}} \nabla ζ = [\begin{matrix} 1 \\ \frac{ζ_{y}}{ζ_{x}} \\ \frac{ζ_{z}}{ζ_{x}} \end{matrix}] = [\begin{matrix} 1 \\ β (x, y) \\ β (x, y) \frac{F_{z}}{F_{y}} \end{matrix}]

is parallel to a gradient vector field

\nabla ζ

⁠. That is, we want the vector field to be perpendicular to its curl (see theorem 5). We have:

\begin{matrix} (\frac{\partial}{\partial x} + β (x, y) \frac{\partial}{\partial y} + β (x, y) \frac{F_{z}}{F_{y}} \frac{\partial}{\partial z}) . curl (\frac{\partial}{\partial x} + β (x, y) \frac{\partial}{\partial y} + β (x, y) \frac{F_{z}}{F_{y}} \frac{\partial}{\partial z}) \\ = β_{y} \frac{F_{z}}{F_{y}} + β {(\frac{F_{z}}{F_{y}})}_{y} - β^{2} {(\frac{F_{z}}{F_{y}})}_{x} . \end{matrix}

The vanishing of the expression above results in a description of

{(\frac{F_{z}}{F_{y}})}_{x}

as the linear combination

{(\frac{F_{z}}{F_{y}})}_{x} = \frac{β_{y}}{β^{2}} \frac{F_{z}}{F_{y}} + \frac{1}{β} {(\frac{F_{z}}{F_{y}})}_{y}

(3.12)

whose coefficients

\frac{β_{y}}{β^{2}} = - {(\frac{1}{β})}_{y}

and

\frac{1}{β}

are independent of

z

⁠. Thus, we are in a situation similar to that of examples 2 and 7, where we encountered identity . The same idea used there could be applied again to obtain PDE constraints: Differentiating equation with respect to

z

results in a linear system:

[\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}] [\begin{matrix} - {(\frac{1}{β})}_{y} \\ \frac{1}{β} \end{matrix}] = [\begin{matrix} {(\frac{F_{z}}{F_{y}})}_{x} \\ {(\frac{F_{z}}{F_{y}})}_{x z} \end{matrix}] .

Assuming the matrix above is nonsingular, Cramer's rule implies

- {(\frac{1}{β})}_{y} = \frac{|\begin{matrix} {(\frac{F_{z}}{F_{y}})}_{x} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{x z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}|}{|\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}|}, \frac{1}{β} = \frac{|\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{x} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{x z} \end{matrix}|}{|\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}|} .

(3.13)

We now arrive at the desired PDE characterization of superpositions . In each of the ratios of determinants appearing in equation , the numerator and denominator are in the form of polynomials of partial derivatives divided by

{(F_{y})}^{4}

⁠. So we introduce the following polynomial expressions:

\begin{matrix} Ψ_{1} & = & {(F_{y})}^{4} |\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}|, \\ Ψ_{2} & = & {(F_{y})}^{4} |\begin{matrix} \frac{F_{z}}{F_{y}} & {(\frac{F_{z}}{F_{y}})}_{x} \\ {(\frac{F_{z}}{F_{y}})}_{z} & {(\frac{F_{z}}{F_{y}})}_{x z} \end{matrix}|, \\ Ψ_{3} & = & {(F_{y})}^{4} |\begin{matrix} {(\frac{F_{z}}{F_{y}})}_{x} & {(\frac{F_{z}}{F_{y}})}_{y} \\ {(\frac{F_{z}}{F_{y}})}_{x z} & {(\frac{F_{z}}{F_{y}})}_{y z} \end{matrix}| . \end{matrix}

(3.14)

Then in view of equation ,

\frac{Ψ_{2}}{Ψ_{1}} = \frac{1}{β}, \frac{Ψ_{3}}{Ψ_{1}} = - {(\frac{1}{β})}_{y} .

(3.15)

Hence

{(\frac{Ψ_{2}}{Ψ_{1}})}_{y} + \frac{Ψ_{3}}{Ψ_{1}} = 0

⁠; furthermore,

{(\frac{Ψ_{2}}{Ψ_{1}})}_{z} = 0

since

β

is independent of

z

⁠:

\begin{matrix} Φ_{1} : = Ψ_{1} {(Ψ_{2})}_{y} - {(Ψ_{1})}_{y} Ψ_{2} + Ψ_{1} Ψ_{3} = 0, Φ_{2} : = Ψ_{1} {(Ψ_{2})}_{z} - {(Ψ_{1})}_{z} Ψ_{2} = 0 . \end{matrix}

(3.16)

Again as in example 7, a continuity argument implies that the algebraic PDEs above are necessary even when the denominator in equation (i.e.,

Ψ_{1}

⁠) is zero. As for the nonvanishing conditions, in view of equations and , we require

F_{y}

to be nonzero as well as

Ψ_{1}

and

Ψ_{2}

(recall that

β \neq 0

⁠):

Ψ_{1} \neq 0, Ψ_{2} \neq 0, F_{y} \neq 0 .

(3.17)

It is easy to see that these conditions are not vacuous for functions of the form . If

F (x, y, z) = {(x y)}^{2} z + z^{3}

⁠, neither

F_{y}

nor the expression

Ψ_{1}

or

Ψ_{2}

is identically zero.

Figure 10:

View large Download slide

An asymmetric tree architecture that computes the superpositions of the form $F (x, y, z) = g (x, f (h (x, y), z))$ ⁠. These are characterized in example 8.

In summary, a special case of conjecture 1 has been verified in this example. A function $F = F (x, y, z)$ of the form 3.11 satisfies the constraints 3.16; conversely, a smooth function satisfying them along with the nonvanishing conditions 3.17 admits a local representation of that form.

3.2 Examples of Functions Computed by Neural Networks

We now switch from trees to examples of PDE constraints for neural networks. The first two examples are concerned with the network illustrated on the left of Figure 11; this is a ResNet with two hidden layers that has

x, y, z, w

as its inputs. The functions it implements are in the form of

F (x, y, z, w) = g (h_{1} (f (x, y), z), h_{2} (f (x, y), w)),

(3.18)

where

f

and

h_{1}, h_{2}

are the functions appearing in the hidden layers.

Figure 11:

View large Download slide

The space of functions computed by the neural network on the left is strictly smaller than that of its TENN on the right. See example 9.

3.2.1 Example 9

On the right of Figure 11, the tree architecture corresponding to the neural network discussed above is illustrated. The functions implemented by this tree are in the form of

F (x, y, z, w) = g (h_{1} (f_{1} (x, y), z), h_{2} (f_{2} (x, y), w)),

(3.19)

which is a form more general than the form of functions computable by the network. In fact, there are PDEs satisfied by the latter class that functions in the former class, equation , do not necessarily satisfy. To see this, observe that for a function

F (x, y, z, w)

of the form , the ratio

\frac{F_{y}}{F_{x}}

coincides with

\frac{f_{y}}{f_{x}}

and is thus independent of

z

and

w

—hence the PDEs

F_{x z} F_{y} = F_{y z} F_{x}

and

F_{x w} F_{y} = F_{y w} F_{x}

⁠. Neither of them holds for the function

F (x, y, z, w) = x y z + (x + y) w

⁠, which is of the form . We deduce that the set of PDE constraints for a network may be strictly larger than that of the corresponding TENN.

3.2.2 Example 10

Here we briefly argue that conjecture 1 holds for the network in Figure 11 (which has two hidden layers). The goal is to obtain PDEs that, given suitable nonvacuous, nonvanishing conditions, characterize smooth functions

F (x, y, z, w)

of the form . We seek a description of the form

g (F_{1} (x, y, z), F_{2} (x, y, w))

of

F (x, y, z, w)

where the trivariate functions

F_{1} (x, y, z)

and

F_{2} (x, y, w)

are superpositions

h_{1} (f (x, y), z)

and

h_{2} (f (x, y), w)

with the same bivariate function

f

appearing in both of them. Invoking the logic that has been used repeatedly in section 3.1,

\nabla F

must be a linear combination of

\nabla F_{1}

and

\nabla F_{2}

⁠. Following example 1, the only restriction on the latter two gradients is

\nabla F_{1} ∥ [\begin{matrix} 1 \\ \frac{{(F_{1})}_{y}}{{(F_{1})}_{x}} = \frac{f_{y}}{f_{x}} \\ α (x, y, z) : = \frac{{(F_{1})}_{z}}{{(F_{1})}_{x}} \\ 0 \end{matrix}], \nabla F_{2} ∥ [\begin{matrix} 1 \\ \frac{{(F_{2})}_{y}}{{(F_{2})}_{x}} = \frac{f_{y}}{f_{x}} \\ 0 \\ β (x, y, w) : = \frac{{(F_{2})}_{w}}{{(F_{2})}_{x}} \end{matrix}];

and as observed in example 9, the ratio

\frac{f_{y}}{f_{x}}

coincides with

\frac{F_{y}}{F_{x}}

⁠. Thus, the existence of a representation of the form is equivalent to the existence of a linear relation such as

[\begin{matrix} F_{x} \\ F_{y} \\ F_{z} \\ F_{w} \end{matrix}] = \frac{F_{z}}{α} [\begin{matrix} 1 \\ \frac{F_{y}}{F_{x}} \\ α (x, y, z) \\ 0 \end{matrix}] + \frac{F_{w}}{β} [\begin{matrix} 1 \\ \frac{F_{y}}{F_{x}} \\ 0 \\ β (x, y, w) \end{matrix}] .

This amounts to the equation

F_{z} (\frac{1}{α}) + F_{w} (\frac{1}{β}) = F_{x} .

Now the idea of examples 2 and 7 applies. As

{(\frac{1}{α})}_{w} = 0

and

{(\frac{1}{β})}_{z} = 0

⁠, applying the operators

\partial_{z}

⁠,

\partial_{w}

⁠, and

\partial_{z w}

to the last equation results in a linear system with four equations and four unknowns:

\frac{1}{α}

⁠,

\frac{1}{β}

⁠,

{(\frac{1}{α})}_{z}

⁠, and

{(\frac{1}{β})}_{w}

⁠. If nonsingular (a nonvanishing condition), the system may be solved to obtain expressions purely in terms of partial derivatives of

F

for the aforementioned unknowns. Now

{(\frac{1}{α})}_{w} = 0

and

{(\frac{1}{β})}_{z} = 0

⁠, along with the equations

F_{x z} F_{y} = F_{y z} F_{x}

⁠,

F_{x w} F_{y} = F_{y w} F_{x}

from example 9, yield four algebraic PDEs characterizing superpositions .

The final example of this section finishes example 3 from the section 1.

3.2.3 Example 11

We go back to example 3 to study PDEs and PDIs satisfied by functions of the form . Absorbing

a^{''}, b^{''}

into inner functions, we can focus on the simpler form:

F (x, t) = σ (f (a x + b t) + g (a^{'} x + b^{'} t)) .

(3.20)

Let us for the time being forget about the outer activation function

σ

⁠. Consider functions such as

G (x, t) = f (a x + b t) + g (a^{'} x + b^{'} t) .

Smooth functions of this form constitute solutions of a second-order linear homogeneous PDE with constant coefficients

U G_{x x} + V G_{x t} + W G_{t t} = 0,

(3.21)

where

(a, b)

and

(a^{'}, b^{'})

satisfy

U A^{2} + V A B + W B^{2} = 0 .

(3.22)

The reason is that when

(a, b)

and

(a^{'}, b^{'})

satisfy equation , the differential operator

U \partial_{x x} + V \partial_{x t} + W \partial_{t t}

can be factorized as

(b \partial_{x} - a \partial_{t}) (b^{'} \partial_{x} - a^{'} \partial_{t})

to a composition of operators that annihilate the linear forms

a x + b t

and

a^{'} x + b^{'} t

⁠. If

(a, b)

and

(a^{'}, b^{'})

are not multiples of each other, they constitute a new coordinate system

(a x + b t, a^{'} x + b^{'} t)

in which the mixed partial derivatives of

F

all vanish; so, at least locally,

F

must be a sum of univariate functions of

a x + b t

and

a^{'} x + b^{'} t

⁠.^⁷ We conclude that assuming

V^{2} - 4 U W > 0

⁠, functions of the form

G (x, t) = f (a x + b t) + g (a^{'} x + b^{'} t)

may be identified with solutions of PDEs of the form . As in example 1, we desire algebraic PDEs purely in terms of

F

and without constants

U, V

⁠, and

W

⁠. One way to do so is to differentiate equation further, for instance:

U G_{x x x} + V G_{x x t} + W G_{x t t} = 0 .

(3.23)

Notice that equations and could be interpreted as

(U, V, W)

being perpendicular to

(G_{x x}, G_{x t}, G_{t t})

and

(G_{x x x}, G_{x x t}, G_{x t t})

⁠. Thus, the cross-product

(G_{x t} G_{x t t} - G_{t t} G_{x x t}, G_{t t} G_{x x x} - G_{x x} G_{x t t}, G_{x x} G_{x x t} - G_{x t} G_{x x x})

of the latter two vectors must be parallel to a constant vector. Under the nonvanishing condition that one of the entries of the cross-product, say the last one, is nonzero, the constancy may be thought of as ratios of the other two components to the last one being constants. The result is a characterization (in the vein of conjecture 1) of functions

G

of the form

f (a x + b t) + g (a^{'} x + b^{'} t)

⁠, which are subjected to

G_{x x} G_{x x t} - G_{x t} G_{x x x} \neq 0

and

\begin{matrix} \frac{G_{x t} G_{x t t} - G_{t t} G_{x x t}}{G_{x x} G_{x x t} - G_{x t} G_{x x x}} and \frac{G_{t t} G_{x x x} - G_{x x} G_{x t t}}{G_{x x} G_{x x t} - G_{x t} G_{x x x}} are constants, \\ {(G_{t t} G_{x x x} - G_{x x} G_{x t t})}^{2} > 4 (G_{x t} G_{x t t} - G_{t t} G_{x x t}) (G_{x x} G_{x x t} - G_{x t} G_{x x x}) . \end{matrix}

(3.24)

Notice that the PDI is not redundant here. For a solution

G = G (x, t)

of Laplace's equation, the fractions from the first line of equation are constants, while on the second line, the left-hand side of the inequality is zero but its right-hand side is

4 {(G_{x t} G_{x t t} - G_{t t} G_{x x t})}^{2} \geq 0

⁠.

Composing

G

with

σ

makes the derivation of PDEs and PDIs imposed on functions of the form even more cumbersome. We provide only a sketch. Under the assumption that the gradient of

F = σ \circ G

is nonzero, the univariate function

σ

admits a local inverse

τ

⁠. Applying the chain rule to

G = τ \circ F

yields

\begin{matrix} G_{x x} & = & τ^{''} (F) {(F_{x})}^{2} + τ^{'} (F) F_{x x}, \\ G_{x t} & = & τ^{''} (F) F_{x} F_{t} + τ^{'} (F) F_{x t}, \\ G_{t t} & = & τ^{''} (F) {(F_{t})}^{2} + τ^{'} (F) F_{t t} . \end{matrix}

Plugging them in the PDE that

G

satisfies results in

τ^{''} (F) (U {(F_{x})}^{2} + V F_{x} F_{t} + W {(F_{t})}^{2}) + τ^{'} (F) (U F_{x x} + V F_{x t} + W F_{t t}) = 0,

or, equivalently,

\frac{U F_{x x} + V F_{x t} + W F_{t t}}{U {(F_{x})}^{2} + V F_{x} F_{t} + W {(F_{t})}^{2}} = - \frac{τ^{''} (F)}{τ^{'} (F)} = - (\frac{τ^{''}}{τ^{'}}) (F) .

(3.25)

It suffices for the ratio

\frac{U F_{x x} + V F_{x t} + W F_{t t}}{U {(F_{x})}^{2} + V F_{x} F_{t} + W {(F_{t})}^{2}}

to be a function of

F

such as

ν (F)

since then

τ

may be recovered as

τ = \int e^{- \int ν}

⁠. Following the discussion at the beginning of section 3, this is equivalent to

\nabla (\frac{U F_{x x} + V F_{x t} + W F_{t t}}{U {(F_{x})}^{2} + V F_{x} F_{t} + W {(F_{t})}^{2}}) ∥ \nabla F .

This amounts to an identity of the form

Φ_{1} (F) U^{2} + Φ_{2} (F) V^{2} + Φ_{3} (F) W^{2} + Φ_{4} (F) U V + Φ_{5} (F) V W + Φ_{6} (F) U W = 0,

where

Φ_{i} (F)

's are complicated nonconstant polynomial expressions of partial derivatives of

F

⁠. In the same way that the parameters

U, V

⁠, and

W

in PDE were eliminated to arrive at equation , one may solve the homogeneous linear system consisting of the identity above and its derivatives in order to derive a six-dimensional vector,

(Ξ_{1} (F), Ξ_{2} (F), Ξ_{3} (F), Ξ_{4} (F), Ξ_{5} (F), Ξ_{6} (F))

(3.26)

of rational expressions of partial derivatives of

F

parallel to the constant vector

(U^{2}, V^{2}, W^{2}, U V, V W, U W) .

(3.27)

The parallelism amounts to a number of PDEs, for example,

Ξ_{1} (F) Ξ_{2} (F) = Ξ_{4} {(F)}^{2}

⁠, and the ratios

\frac{Ξ_{i} (F)}{Ξ_{j} (F)}

must be constant because they coincide with the ratios of components of equation . Moreover,

V^{2} - 4 U W > 0

implies

(\frac{V^{2}}{U W} - 4) \frac{V^{2}}{U W} \geq 0

⁠. Replacing with the corresponding ratios of components of equation , we obtain the PDI

(Ξ_{2} (F) - 4 Ξ_{6} (F)) Ξ_{2} (F) Ξ_{6} (F) \geq 0,

which must be satisfied by any function of the form .

3.3 Examples of Polynomial Neural Networks

The superpositions we study in this section are constructed out of polynomials. Again, there are two different regimes to discuss: composing general polynomial functions of low dimensionality or composing polynomials of arbitrary dimensionality but in the simpler form of

y \mapsto σ (〈 w, y 〉)

where the activation function

σ

is a polynomial of a single variable. The latter regime deals with polynomial neural networks. Different aspects of such networks have been studied in the literature (Du & Lee, 2018; Soltanolkotabi et al., 2018; Venturi et al., 2018; Kileel et al., 2019). In the spirit of this article, we are interested in the spaces formed by such polynomial superpositions. Bounding the total degree of polynomials from the above, these functional spaces are subsets of an ambient polynomial space, say, the space

{Poly}_{d, n}

of real polynomials

P (x_{1}, \dots, x_{n})

of total degree at most

d

⁠, which is an affine space of dimension

(\binom{d + n}{n})

⁠. By writing a polynomial

P (x_{1}, \dots, x_{n})

of degree

d

as

P (x_{1}, x_{2}, \dots, x_{n}) = \sum_{\binom{a_{1}, a_{2}, \dots, a_{n} \geq 0}{a_{1} + a_{2} + \dots + a_{n} \leq d}} c_{a_{1}, a_{2}, \dots, a_{n}} x_{1}^{a_{1}} x_{2}^{a_{2}} \dots x_{n}^{a_{n}},

(3.28)

the coefficients

c_{a_{1}, a_{2}, \dots, a_{n}}

provide a natural coordinate system on

{Poly}_{d, n}

⁠. Associated with a neural network

N

that receives

x_{1}, \dots, x_{n}

as its inputs, there are polynomial functional spaces for any degree

d

that lie in the ambient space

{Poly}_{d, n}

⁠:

The subset $F_{d} (N)$ of ${Poly}_{d, n}$ consisting of polynomials $P (x_{1}, \dots, x_{n})$ of total degree at most $d$ that can be computed by $N$ via assigning real polynomial functions to its neurons
The smaller subset $F_{d}^{act} (N)$ of ${Poly}_{d, n}$ consisting of polynomials $P (x_{1}, \dots, x_{n})$ of total degree at most $d$ that can be computed by $N$ via assigning real polynomials of the form $y \mapsto σ (〈 w, y 〉)$ to the neurons where $σ$ is a polynomial activation function

In general, subsets $F_{d} (N)$ and $F_{d}^{act} (N)$ of ${Poly}_{d, n}$ are not closed in the algebraic sense (see remark 8). Therefore, one may consider their Zariski closures $V_{d} (N)$ and $V_{d}^{act} (N)$ ⁠, that is, the smallest subsets defined as zero loci of polynomial equations that contain them. We shall call $V_{d} (N)$ and $V_{d}^{act} (N)$ the functional varieties associated with $N$ ⁠. Each of the subsets $V_{d} (N)$ and $V_{d}^{act} (N)$ of ${Poly}_{d, n}$ could be described with finitely many polynomial equations in terms of $c_{a_{1}, a_{2}, \dots, a_{n}}$ 's. The PDE constraints from section 2 provide nontrivial examples of equations satisfied on the functional varieties: In any degree $d$ ⁠, substituting equation 3.28 in an algebraic PDE that smooth functions computed by $N$ must obey results in equations in terms of the coefficients that are satisfied at any point of $F_{d} (N)$ or $F_{d}^{act} (N)$ and hence at the points of $V_{d} (N)$ or $V_{d}^{act} (N)$ ⁠. This will be demonstrated in example 12 and results in the following corollary to theorems 1 and 2.

Corollary 2.

Let $N$ be a neural network whose inputs are labeled by the coordinate functions $x_{1}, \dots, x_{n}$ ⁠. Then there exist nontrivial polynomials on affine spaces ${Poly}_{d, n}$ that are dependent only on the topology of $N$ and become zero on functional varieties $V_{d}^{act} (N) \subset {Poly}_{d, n}$ ⁠. The same holds for functional varieties $V_{d} (N)$ provided that the number of inputs to each neuron of $N$ is less than $n$ ⁠.

Proof.

The proof immediately follows from theorem 2 (in the case of

V_{d}^{act} (N)

⁠) and from theorem 1 (in the case of

V_{d} (N)

⁠). Substituting a polynomial

P (x_{1}, \dots, x_{n})

in a PDE constraint

Φ (P_{x_{1}}, \dots, P_{x_{n}}, P_{x_{1}^{2}}, P_{x_{1} x_{2}}, \dots, P_{x^{α}}, \dots) = 0

that these theorems suggest for

N

and equating the coefficient of a monomial

x_{1}^{a_{1}} x_{2}^{a_{2}} \dots x_{n}^{a_{n}}

with zero results in a polynomial equation in ambient polynomial spaces that must be satisfied on the associated functional varieties.

3.3.1 Example 12

Let

N

be a rooted tree

T

with distinct inputs

x_{1}, \dots, x_{n}

⁠. Constraints of the form

F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}

are not only necessary conditions for a smooth function

F = F (x_{1}, \dots, x_{n})

to be computable by

T

⁠; but by the virtue of theorem 3, they are also sufficient for the existence of a local representation of

F

on

T

if suitable nonvanishing conditions are satisfied. An interesting feature of this setting is that when

F

is a polynomial

P = P (x_{1}, \dots, x_{n})

⁠, one can relax the nonvanishing conditions; and

P

actually admits a global representation as a composition of polynomials if it satisfies the characteristic PDEs (Farhoodi et al., 2019, proposition 4). The basic idea is that if

P

is locally written as a superposition of smooth functions according to the hierarchy provided by

T

⁠, then comparing the Taylor series shows that the constituent parts of the superposition could be chosen to be polynomials as well. Now

P

and such a polynomial superposition must be the same since they agree on a nonempty open set. Consequently, each

F_{d} (N)

coincides with its closure

V_{d} (N)

and can be described by equations of the form

P_{x_{i} x_{k}} P_{x_{j}} = P_{x_{j} x_{k}} P_{x_{i}}

in the polynomial space. Substituting an expression of the form

P (x_{1}, \dots, x_{n}) = \sum_{a_{1}, \dots, a_{n} \geq 0} c_{a_{1}, \dots, a_{n}} x_{1}^{a_{1}} \dots x_{n}^{a_{n}}

in

P_{x_{i} x_{k}} P_{x_{j}} - P_{x_{j} x_{k}} P_{x_{i}} = 0

and equating the coefficient of a monomial

x_{1}^{a_{1}} \dots x_{n}^{a_{n}}

with zero yields

\sum_{\binom{a_{i}^{'} + a_{i}^{''} = a_{i} + 1, a_{j}^{'} + a_{j}^{''} = a_{j} + 1, a_{k}^{'} + a_{k}^{''} = a_{k} + 1}{a_{s}^{'} + a_{s}^{''} = a_{s} \forall s \in {1, \dots, n} - {i, j, k}}} a_{k}^{'} (a_{i}^{'} a_{j}^{''} - a_{j}^{'} a_{i}^{''}) c_{a_{1}^{'}, \dots, a_{n}^{'}} c_{a_{1}^{''}, \dots, a_{n}^{''}} = 0 .

(3.29)

We deduce that equations written for

a_{1}, \dots, a_{n} \geq 0

and for triples

(i, j, k)

with the property that

x_{k}

is separated from

x_{i}

and

x_{j}

by a subtree of

T

(as in theorem 3) describe the functional varieties associated with

T

⁠. In a given degree

d

⁠, to obtain equations describing

T

in

{Poly}_{d, n}

⁠, one should set any

c_{b_{1}, \dots, b_{n}}

with

b_{1} + \dots + b_{n} > d

to be zero in equation . No such a coefficient occurs if

d \geq a_{1} + \dots + a_{n} + 3

⁠, and thus for

d

large enough, equation defines an equation in

{Poly}_{d, n}

as is.

Similarly, theorem 7 can be used to write equations for $F_{d}^{act} (N) = V_{d}^{act} (N)$ ⁠. In that situation, a new family of equations corresponding to equation 1.20 emerges that are expected to be extremely complicated.

3.3.2 Example 13

Let $N$ be the neural network appearing in Figure 4. The functional space $F_{d}^{act} (N)$ is formed by polynomials $P (x, t)$ of total degree at most $d$ that are in the form of $σ (f (a x + b t) + g (a^{'} x + b^{'} t))$ ⁠. By examining the Taylor expansions, it is not hard to see that if $P (x, t)$ is written in this form for univariate smooth functions $σ$ ⁠, $f$ ⁠, and $g$ ⁠, then these functions could be chosen to be polynomials. Therefore, in any degree $d$ ⁠, our characterization of superpositions of this form in example 11 in terms of PDEs and PDIs results in polynomial equations and inequalities that describe a Zariski open subset of $F_{d}^{act} (N)$ which is the complement of the locus where the nonvanishing conditions fail. The inequalities disappear after taking the closure, so $V_{d}^{act} (N)$ is strictly larger than $F_{d}^{act} (N)$ here.

Remark 8.

The emergence of inequalities in describing the functional spaces, as observed in example 13, is not surprising due to the Tarski–Seidenberg theorem (see Coste, 2000), which implies that the image of a polynomial map between real varieties (i.e., a map whose components are polynomials) is semialgebraic; that is, it could be described as a union of finitely many sets defined by polynomial equations and inequalities. To elaborate, fix a neural network architecture

N

⁠. Composing polynomials of bounded degrees according to the hierarchy provided by

N

yields polynomial superpositions lying in

F_{D} (N)

for

D

sufficiently large. The composition thus amounts to a map

{Poly}_{d_{1}, n_{1}} \times \dots \times {Poly}_{d_{N}, n_{N}} \to {Poly}_{D, n},

where, on the left-hand side, the polynomials assigned to the neurons of

N

appear, and

D ≫ d_{1}, \dots, d_{n}

⁠. The image, a subset of

F_{D} (N)

⁠, is semialgebraic and thus admits a description in terms of finitely many polynomial equations and inequalities. The same logic applies to the regime of activation functions too; the map just mentioned must be replaced with

R^{C} \times {Poly}_{d_{1}, 1} \times \dots \times {Poly}_{d_{N}, 1} \to {Poly}_{D, n}

whose image lies in

F_{D}^{act} (N)

⁠, and its domain is the Cartesian product of spaces of polynomial activation functions assigned to the neurons by the space

R^{C}

of weights assigned to the connections of the network.

4 PDE Characterization of Tree Functions

Building on the examples of the previous section, we prove theorems 3 and 4. This will establish conjecture 1 for tree architectures with distinct inputs.

Proof of Theorem 3.

The necessity of the constraints from equation follows from example 1. As demonstrated in Figure 12, picking three of variables

x_{i} = x

⁠,

x_{j} = y

⁠, and

x_{k} = z

where the former two are separated from the latter by a subtree and taking the rest of variables to be constant, we obtain a superposition of the form

F (x, y, z) = g (f (x, y), z)

studied in example 1; it should satisfy

F_{x z} F_{y} = F_{y z} F_{x}

or, equivalently, equation .

Figure 12:

View large Download slide

The necessity of constraint 1.18 in theorem 3 follows from the case of trivariate tree functions discussed in example 1. Choosing three of the variables (red leaves) and fixing the rest (gray leaves) results in a superposition of the form $g (f (x, y), z)$ that must obey constraint 1.4.

We induct on the number of variables, which coincides with the number of leaves, to prove the sufficiency of constraint and the nonvanishing conditions in theorem 3 for the existence of a local implementation, in the form of a superposition of functions of lower arity, on the tree architecture in hand. Consider a rooted tree

T

with

n

leaves labeled by the coordinate functions

x_{1}, \dots, x_{n}

⁠. The inductive step is illustrated in Figure 13. Removing the root results in a number of smaller trees

T_{1}, \dots, T_{l}

and a number of single vertices^⁸ corresponding to the leaves adjacent to the root of

T

⁠. By renumbering

x_{1}, \dots, x_{n}

one may write the leaves as

\begin{matrix} x_{1}, \dots, x_{m_{1}}; x_{m_{1} + 1}, \dots, x_{m_{1} + m_{2}}; \dots; x_{m_{1} + \dots + m_{l - 1} + 1}, \dots, x_{m_{1} + \dots + m_{l}}; \\ x_{m_{1} + \dots + m_{l} + 1}; \dots; x_{n}, \end{matrix}

(4.1)

where

x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s - 1} + m_{s}}

(1 \leq s \leq l)

are the leaves of the subtree

T_{s}

while

x_{m_{1} + \dots + m_{l} + 1}

through

x_{n}

are the leaves adjacent to the root of

T

⁠. The goal is to write

F (x_{1}, \dots, x_{n})

as

\begin{matrix} g (G_{1} (x_{1}, \dots, x_{m_{1}}), \dots, G_{l} (x_{m_{1} + \dots + m_{l - 1} + 1}, \dots, x_{m_{1} + \dots + m_{l}}), x_{m_{1} + \dots + m_{l} + 1}, \dots, x_{n}), \end{matrix}

(4.2)

where each smooth function,

G_{s} (x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s - 1} + m_{s}}),

satisfies the constraints coming from

T_{s}

and thus, by invoking the induction hypothesis, is computable by the tree

T_{s}

⁠. Following the discussion before theorem 5, it suffices to express

\nabla F

as a linear combination of the gradients

\nabla G_{1}, \dots, \nabla G_{l}, \nabla x_{m_{1} + \dots + m_{l} + 1}, \dots, \nabla x_{n}

⁠. The nonvanishing conditions in theorem 3 require the first-order partial derivative with respect to at least one of the leaves of each

T_{s}

to be nonzero; we may assume

F_{x_{m_{1} + \dots + m_{s - 1} + 1}} \neq 0

without any loss of generality. We should have

\begin{matrix} \nabla F = {[F_{x_{1}} \dots F_{x_{m_{1}}} \dots F_{x_{m_{1} + \dots + m_{l - 1} + 1}} \dots F_{x_{m_{1} + \dots + m_{l}}} F_{x_{m_{1} + \dots + m_{l} + 1}} \dots F_{x_{n}}]}^{T} \\ = \sum_{s = 1}^{l} F_{x_{m_{1} + \dots + m_{s - 1} + 1}} {[\overset{m_{1} + \dots + m_{s - 1}}{\overset{⏞}{0 \dots 0}} 1 \frac{F_{x_{m_{1} + \dots + m_{s - 1} + 2}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}} \dots \frac{F_{x_{m_{1} + \dots + m_{s - 1} + m_{s}}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}} \overset{n - (m_{1} + \dots + m_{s})}{\overset{⏞}{0 \dots 0}}]}^{T} \\ + F_{x_{m_{1} + \dots + m_{l} + 1}} \frac{\partial}{\partial x_{m_{1} + \dots + m_{l} + 1}} + \dots + F_{x_{n}} \frac{\partial}{\partial x_{n}} \\ \in Span {\nabla G_{1} (x_{1}, \dots, x_{m_{1}}), \dots, \nabla G_{l} (x_{m_{1} + \dots + m_{l - 1} + 1}, \dots, x_{m_{1} + \dots + m_{l}}), \\ \nabla x_{m_{1} + \dots + m_{l} + 1}, \dots, \nabla x_{n}} . \end{matrix}

In the expressions above, the vector

{[1 \frac{F_{x_{m_{1} + \dots + m_{s - 1} + 2}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}} \dots \frac{F_{x_{m_{1} + \dots + m_{s - 1} + m_{s}}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}}]}^{T}

(which is of size

m_{s}

⁠) is dependent only on the variables

x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s}}

⁠, which are the leaves of

T_{s}

⁠: any other leaf

x_{k}

is separated from them by the subtree

T_{s}

of

T

⁠, and, hence, for any leaf

x_{i}

with

m_{1} + \dots + m_{s - 1} < i \leq m_{1} + \dots + m_{s}

⁠, we have

{(\frac{F_{x_{i}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}})}_{x_{k}} = 0

due to the simplified form of equation . To finish the proof, one should establish the existence of functions

G_{s} (x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s}})

appearing in equation ; that is,

{[1 \frac{F_{x_{m_{1} + \dots + m_{s - 1} + 2}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}} \dots \frac{F_{x_{m_{1} + \dots + m_{s - 1} + m_{s}}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}}]}^{T}

should be shown to be parallel to a gradient vector field

\nabla G_{s}

⁠. Notice that the induction hypothesis would be applicable to

G_{s}

since any ratio

\frac{{(G_{s})}_{x_{i}}}{{(G_{s})}_{x_{j}}}

of partial derivatives is the same as the corresponding ratio of partial derivatives of

F

⁠. Invoking theorem 5, to prove the existence of

G_{s}

⁠, we should verify that the 1-form

ω_{s} : = \sum_{i = m_{1} + \dots + m_{s - 1} + 1}^{m_{1} + \dots + m_{s - 1} + m_{s}} \frac{F_{x_{i}}}{F_{x_{m_{1} + \dots + m_{s - 1} + 1}}} d x_{i} (1 \leq s \leq l)

satisfies

ω_{s} \land d ω_{s} = 0

⁠. We finish the proof by showing this in the case of

s = 1

⁠; other cases are completely similar. We have

\begin{matrix} ω_{1} \land d ω_{1} & = & (\sum_{i = 1}^{m_{1}} \frac{F_{x_{i}}}{F_{x_{1}}} d x_{i}) \land (\sum_{j = 1}^{m_{1}} d (\frac{F_{x_{j}}}{F_{x_{1}}}) \land d x_{j}) \\ = & (\sum_{i = 1}^{m_{1}} \frac{F_{x_{i}}}{F_{x_{1}}} d x_{i}) \land (\sum_{j = 1}^{m_{1}} (\sum_{k = 1}^{m_{1}} {(\frac{F_{x_{j}}}{F_{x_{1}}})}_{x_{k}} d x_{k}) \land d x_{j}) \\ = & \sum_{i, j, k \in {1, \dots, m_{1}}} [\frac{F_{x_{i}} F_{x_{j} x_{k}}}{{(F_{x_{1}})}^{2}} - \frac{F_{x_{i}} F_{x_{j}} F_{x_{1} x_{k}}}{{(F_{x_{1}})}^{3}}] d x_{i} \land d x_{k} \land d x_{j} \\ = & (\sum_{i = 1}^{m_{1}} \frac{F_{x_{i}}}{{(F_{x_{1}})}^{2}} d x_{i}) \land (\sum_{j, k \in {1, \dots, m_{1}}} F_{x_{j} x_{k}} d x_{k} \land d x_{j}) \\ + (\sum_{i, j \in {1, \dots, m_{1}}} F_{x_{i}} F_{x_{j}} d x_{i} \land d x_{j}) \land (\sum_{k = 1}^{m_{1}} \frac{F_{x_{1} x_{k}}}{{(F_{x_{1}})}^{3}} d x_{k}) . \end{matrix}

The last two terms are zero because, in the parentheses, the 2-forms

\sum_{j, k \in {1, \dots, m_{1}}} F_{x_{j} x_{k}} d x_{j} \land d x_{k}, \sum_{i, j \in {1, \dots, m_{1}}} F_{x_{i}} F_{x_{j}} d x_{i} \land d x_{j},

are zero since interchanging

j

and

k

or

i

and

j

in the summations results in the opposite of the original differential form.

Figure 13:

View large Download slide

The inductive step in the proof of theorem 3. The removal of the root of $T$ results in a number of smaller rooted trees along with single vertices that were the leaves adjacent to the root of $T$ (if any).

Remark 9.

The formulation of theorem 3 in Farhoodi et al. (2019) is concerned with analytic functions and binary trees. The proof presented above follows the same inductive procedure but utilizes theorem 5 instead of Taylor expansions. Of course, theorem 5 remains valid in the analytic category, so the tree representation of $F$ constructed in the proof here consists of analytic functions if $F$ is analytic. An advantage of working with analytic functions is that in certain cases, the nonvanishing conditions may be relaxed. For instance, if in example 1 the function $F (x, y, z)$ satisfying equation 1.4 is analytic, it admits a local representation of the form 1.1, while if $F$ is only smooth, at least one of the conditions $F_{x} \neq 0$ of $F_{y} \neq 0$ is required. (See Farhoodi et al., 2019, sec. 5.1 and 5.3, for details.)

Proof of Theorem 4.

Establishing the necessity of constraints and is straightforward. An implementation of a smooth function

F = F (x_{1}, \dots, x_{n})

on the tree

T

is in a form such as

\begin{matrix} σ (\dots (\tilde{w} . \tilde{σ} (w_{1} . τ_{1} (\dots ({\tilde{w}}_{1} . {\tilde{τ}}_{1} (c x_{i} + \dots) + {\tilde{\tilde{w}}}_{1} . {\tilde{\tilde{τ}}}_{1} (c^{'} x_{i^{'}} + \dots) + \dots) \dots) \\ + w_{2} . τ_{2} (\dots ({\tilde{w}}_{2} . {\tilde{τ}}_{2} (d x_{j} + \dots) + {\tilde{\tilde{w}}}_{2} . {\tilde{\tilde{τ}}}_{2} (d^{'} x_{j^{'}} + \dots) + \dots) \dots) \\ + w_{3} . τ_{3} (\dots) + \dots) + \dots) \dots) \end{matrix}

(4.3)

for appropriate activation functions and weights. In the expression above, variables

x_{s}

appearing in

\begin{matrix} \tilde{σ} (w_{1} . τ_{1} (\dots ({\tilde{w}}_{1} . {\tilde{τ}}_{1} (c x_{i} + \dots) + {\tilde{\tilde{w}}}_{1} . {\tilde{\tilde{τ}}}_{1} (c^{'} x_{i^{'}} + \dots) + \dots) \dots) \\ + w_{2} . τ_{2} (\dots ({\tilde{w}}_{2} . {\tilde{τ}}_{2} (d x_{j} + \dots) + {\tilde{\tilde{w}}}_{2} . {\tilde{\tilde{τ}}}_{2} (d^{'} x_{j^{'}} + \dots) + \dots) \dots) \\ + w_{3} . τ_{3} (\dots) + \dots) \end{matrix}

are the leaves of the smallest (full) subtree of

T

in which both

x_{i}

and

x_{j}

appear as leaves. Denoting this subtree by

\tilde{T}

⁠, the activation function applied at the root of

\tilde{T}

is

\tilde{σ}

⁠, and the subtrees emanating from the root of

\tilde{T}

⁠, which we write as

\tilde{T_{1}}, \tilde{T_{2}}, \tilde{T_{3}}, \dots

⁠, have

τ_{1}, τ_{2}, τ_{3}, \dots

assigned to their roots. Here,

\tilde{T_{1}}

and

\tilde{T_{2}}

contain

x_{i}

and

x_{j}

⁠, respectively, and are the largest (full) subtrees that have exactly one of

x_{i}

and

x_{j}

⁠. To verify equation , notice that

\frac{F_{x_{i}}}{F_{x_{j}}}

is proportional to

\begin{matrix} \frac{τ_{1}^{'} (\dots ({\tilde{w}}_{1} . {\tilde{τ}}_{1} (c x_{i} + \dots) + {\tilde{\tilde{w}}}_{1} . {\tilde{\tilde{τ}}}_{1} (c^{'} x_{i^{'}} + \dots) + \dots) \dots) \dots {\tilde{τ}}_{1}^{'} (c x_{i} + \dots)}{τ_{2}^{'} (\dots ({\tilde{w}}_{2} . {\tilde{τ}}_{2} (d x_{j} + \dots) + {\tilde{\tilde{w}}}_{2} . {\tilde{\tilde{τ}}}_{2} (d^{'} x_{j^{'}} + \dots) + \dots) \dots) \dots {\tilde{τ}}_{2}^{'} (d x_{j} + \dots)} \end{matrix}

(4.4)

with the constant of proportionality being a quotient of two products of certain weights of the network. The ratio is dependent only on those variables that appear as leaves of

\tilde{T_{1}}

and

\tilde{T_{2}}

⁠, so

{(\frac{F_{x_{i}}}{F_{x_{j}}})}_{x_{k}} = 0 \Leftrightarrow F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}

unless there is a subtree of

T

containing the leaf

x_{k}

and exactly one of

x_{i}

or

x_{j}

(which forcibly will be a subtree of

\tilde{T_{1}}

or

\tilde{T_{2}}

⁠). Before switching to constraint , we point out that the description of

F

in equation assumes that the leaves

x_{i}

and

x_{j}

are not siblings. If they are,

F

may be written as

σ (\dots (\tilde{w} . \tilde{σ} (w . τ (c x_{i} + d x_{j} + \dots) + \dots) + \dots) \dots),

in which case,

\frac{F_{x_{i}}}{F_{x_{j}}} = \frac{c}{d}

is a constant and hence equation holds for all

1 \leq k \leq n

⁠. To finish the proof of necessity of the constraints introduced in theorem 7, consider the fraction from equation , which is a multiple of

\frac{F_{x_{i}}}{F_{x_{j}}}

⁠. This has a description as a product of a function of

x_{i}, x_{i^{'}}, \dots

(leaves of

\tilde{T_{1}}

⁠) by a function of

x_{j}, x_{j^{'}}, \dots

(leaves of

\tilde{T_{2}}

⁠). Lemma 4 now implies that for any leaf

x_{i^{'}}

of

\tilde{T_{1}}

and any leaf

x_{j^{'}}

of

\tilde{T_{2}}

⁠,

{(\frac{{(\frac{F_{x_{i}}}{F_{x_{j}}})}_{x_{i^{'}}}}{\frac{F_{x_{i}}}{F_{x_{j}}}})}_{x_{j^{'}}} = 0;

hence, the simplified form, equation of .

We induct on the number of leaves to prove the sufficiency of constraints 1.19 and 1.20 (accompanied by suitable nonvanishing conditions) for the existence of a tree implementation of a smooth function $F = F (x_{1}, \dots, x_{n})$ as a composition of functions of the form 1.12. Given a rooted tree $T$ with $n$ leaves labeled by $x_{1}, \dots, x_{n}$ ⁠, the inductive step has two cases demonstrated in Figures 14 and 15:

There are leaves, say, $x_{m + 1}, \dots, x_{n}$ ⁠, directly adjacent to the root of $T$ ⁠; their removal results in a smaller tree $T^{'}$ with leaves $x_{1}, \dots, x_{m}$ (see Figure 14). The goal is to write $F (x_{1}, \dots, x_{n})$ as
$σ (G (x_{1}, \dots, x_{m}) + c_{m + 1} x_{m + 1} + \dots + c_{n} x_{n}),$
(4.5)
with $G$ satisfying appropriate constraints that, invoking the induction hypothesis, guarantee that $G$ is computable by $T^{'}$ ⁠.
Figure 14:
View large Download slide
The first case of the inductive step in the proof of theorem 4. The removal of the leaves directly connected to the root of $T$ results in a smaller rooted tree.
Figure 14:
View large Download slide
The first case of the inductive step in the proof of theorem 4. The removal of the leaves directly connected to the root of $T$ results in a smaller rooted tree.
Close modal
Figure 15:
View large Download slide
The second case of the inductive step in the proof of theorem 4. There is no leaf directly connected to the root of $T$ ⁠. Separating one of the rooted subtrees adjacent to the root results in two smaller rooted trees.
Figure 15:
View large Download slide
The second case of the inductive step in the proof of theorem 4. There is no leaf directly connected to the root of $T$ ⁠. Separating one of the rooted subtrees adjacent to the root results in two smaller rooted trees.
Close modal
There is no leaf adjacent to the root of $T$ ⁠, but there are smaller subtrees. Denote one of them with $T_{2}$ and show its leaves by $x_{m + 1}, \dots, x_{n}$ ⁠. Removing this subtree results in a smaller tree $T_{1}$ with leaves $x_{1}, \dots, x_{m}$ (see Figure 15). The goal is to write $F (x_{1}, \dots, x_{n})$ as
$σ (G_{1} (x_{1}, \dots, x_{m}) + G_{2} (x_{m + 1}, \dots, x_{n})),$
(4.6)
with $G_{1}$ and $G_{2}$ satisfying constraints corresponding to $T_{1}$ and $T_{2}$ ⁠, and hence may be implemented on these trees by invoking the induction hypothesis.

Following the discussion at the beginning of section 3, $F$ may be locally written as a function of another function with nonzero gradient if the gradients are parallel. This idea has been frequently used so far, but there is a twist here: we want such a description of $F$ to persist on the box-like region $B$ that is the domain of $F$ ⁠. Lemma 2 resolves this issue. The tree function in the argument of $σ$ in either of equation 4.5 or 4.6, which here we denote by $\tilde{F}$ ⁠, shall be constructed below by invoking the induction hypothesis, so $\tilde{F}$ is defined at every point of $B$ ⁠. Besides, our description of $\nabla \tilde{F}$ below (cf. equations 4.7 and 4.9) readily indicates that just like $F$ ⁠, it satisfies the nonvanishing conditions of theorem 7. Applying lemma 2 to $\tilde{F}$ ⁠, any level set $\{x \in B | \tilde{F} (x) = c\}$ is connected, and $\tilde{F}$ can be extended to a coordinate system $(\tilde{F}, F_{2}, \dots, F_{n})$ for $B$ ⁠. Thus, $F$ ⁠, whose partial derivatives with respect to other coordinate functions vanish, realizes precisely one value on any coordinate hypersurface $\{x \in B | \tilde{F} (x) = c\}$ ⁠. Setting $σ (c)$ to be the aforementioned value of $F$ defines a function $σ$ with $F = σ (\tilde{F})$ ⁠. After this discussion on the domain of definition of the desired representation of $F$ ⁠, we proceed with constructing $\tilde{F} = \tilde{F} (x_{1}, \dots, x_{n})$ as either $G (x_{1}, \dots, x_{m}) + c_{m + 1} x_{m + 1} + \dots + c_{n} x_{n}$ in the case of equation 4.5 or as $G (x_{1}, \dots, x_{m}) + G_{2} (x_{m + 1}, \dots, x_{n})$ in the case of equation 4.6.

In the case of equation , assuming that, as theorem 7 requires, one of the partial derivatives

F_{x_{m + 1}}, \dots, F_{x_{n}}

⁠, example

F_{x_{n}}

⁠, is nonzero, we should have

\begin{matrix} \nabla F & = & {[F_{x_{1}} \dots F_{x_{m}} F_{x_{m + 1}} \dots F_{x_{n - 1}} F_{x_{n}}]}^{T} ∥ {[\frac{F_{x_{1}}}{F_{x_{n}}} \dots \frac{F_{x_{m}}}{F_{x_{n}}} \frac{F_{x_{m + 1}}}{F_{x_{n}}} \dots \frac{F_{x_{n - 1}}}{F_{x_{n}}} 1]}^{T} \\ = & {[G_{x_{1}} \dots G_{x_{m}} c_{m + 1} \dots c_{n - 1} 1]}^{T} \\ = & \nabla (G (x_{1}, \dots, x_{m}) + c_{m + 1} x_{m + 1} + \dots + c_{n - 1} x_{n - 1} + x_{n}) . \end{matrix}

(4.7)

Here, each ratio

\frac{F_{x_{j}}}{F_{x_{n}}}

where

m < j \leq n

must be a constant, which we show by

c_{j}

⁠, due to the simplified form equation of equation : the only (full) subtree of

T

containing either

x_{j}

or

x_{n}

is the whole tree since these leaves are adjacent to the root of

T

⁠. On the other hand,

{[\frac{F_{x_{1}}}{F_{x_{n}}} \dots \frac{F_{x_{m}}}{F_{x_{n}}}]}^{T}

appearing in equation is a gradient vector field of the form

\nabla G (x_{1}, \dots, x_{m})

again as a by-product of equations and : each ratio

\frac{F_{x_{i}}}{F_{x_{n}}}

where

1 \leq i \leq m

is independent of

x_{m + 1}, \dots, x_{n}

by the same reasoning as above; and this vector function of

(x_{1}, \dots, x_{m})

is integrable because for any

1 \leq i, i^{'} \leq m

⁠,

{(\frac{F_{x_{i}}}{F_{x_{n}}})}_{x_{i^{'}}} = {(\frac{F_{x_{i^{'}}}}{F_{x_{n}}})}_{x_{i}} \Leftrightarrow F_{x_{i} x_{n}} F_{x_{i^{'}}} = F_{x_{i^{'}} x_{n}} F_{x_{i}} .

Hence, such a

G (x_{1}, \dots, x_{m})

exists; moreover, it satisfies constraints from the inductions hypothesis since any ratio

\frac{G_{x_{j}}}{G_{x_{j^{'}}}}

coincides with the corresponding ratio of partial derivatives of

F

⁠, a function assumed to satisfy equations and .

Next, in the second case of the inductive step, let us turn to equation . The nonvanishing conditions of theorem 7 require a partial derivative among

F_{x_{1}}, \dots, F_{x_{m}}

and also a partial derivative among

F_{x_{m + 1}}, \dots, F_{x_{n}}

to be nonzero. Without any loss of generality, we assume

F_{x_{1}} \neq 0

and

F_{x_{n}} \neq 0

⁠. We want to apply lemma 4 to split the ratio

\frac{F_{x_{1}}}{F_{x_{n}}} \neq 0

as

\frac{F_{x_{1}}}{F_{x_{n}}} = β (x_{1}, \dots, x_{m}) \frac{1}{γ} (x_{m + 1}, \dots, x_{n}) = \frac{β (x_{1}, \dots, x_{m})}{γ (x_{m + 1}, \dots, x_{n})} .

(4.8)

To do so, it needs to be checked that

{(\frac{{(\frac{F_{x_{1}}}{F_{x_{n}}})}_{x_{i}}}{\frac{F_{x_{1}}}{F_{x_{n}}}})}_{x_{j}} = 0

for any two indices

1 \leq i \leq m

and

m < j \leq n

⁠. This is the content of equation , or its simplified form, equation , when

x_{i}

belongs to the same maximal subtree of

T

adjacent to the root that has

x_{1}

and holds for other choices of

x_{i} \in {x_{1}, \dots, x_{m}}

too since in that situation, by the simplified form equation , of equation , the derivative

{(\frac{F_{x_{1}}}{F_{x_{n}}})}_{x_{i}}

must be zero because

x_{1}

⁠,

x_{i}

⁠, and

x_{n}

belong to different maximal subtrees of

T

⁠. Next, the gradient of

F

could be written as

\begin{matrix} \nabla F & = & {[F_{x_{1}} F_{x_{2}} \dots F_{x_{m}} F_{x_{m + 1}} \dots F_{x_{n - 1}} F_{x_{n}}]}^{T} \\ ∥ {[\frac{F_{x_{1}}}{F_{x_{n}}} \frac{F_{x_{2}}}{F_{x_{n}}} \dots \frac{F_{x_{m}}}{F_{x_{n}}} \frac{F_{x_{m + 1}}}{F_{x_{n}}} \dots \frac{F_{x_{n - 1}}}{F_{x_{n}}} 1]}^{T} \\ = & {[\frac{F_{x_{1}}}{F_{x_{n}}} \frac{F_{x_{2}}}{F_{x_{1}}} . \frac{F_{x_{1}}}{F_{x_{n}}} \dots \frac{F_{x_{m}}}{F_{x_{1}}} . \frac{F_{x_{1}}}{F_{x_{n}}} \frac{F_{x_{m + 1}}}{F_{x_{n}}} \dots \frac{F_{x_{n - 1}}}{F_{x_{n}}} 1]}^{T} \\ = & \frac{F_{x_{1}}}{F_{x_{n}}} {[1 \frac{F_{x_{2}}}{F_{x_{1}}} \dots \frac{F_{x_{m}}}{F_{x_{1}}} \overset{n - m}{\overset{⏞}{0 \dots 0}}]}^{T} + {[\overset{m}{\overset{⏞}{0 \dots 0}} \frac{F_{x_{m + 1}}}{F_{x_{n}}} \dots \frac{F_{x_{n - 1}}}{F_{x_{n}}} 1]}^{T} . \end{matrix}

Combining with equation :

\begin{matrix} \nabla F ∥ {[β (x_{1}, \dots, x_{m}) β (x_{1}, \dots, x_{m}) . \frac{F_{x_{2}}}{F_{x_{1}}} \dots β (x_{1}, \dots, x_{m}) . \frac{F_{x_{m}}}{F_{x_{1}}} \overset{n - m}{\overset{⏞}{0 \dots 0}}]}^{T} \\ + [\overset{m}{\overset{⏞}{0 \dots 0}} γ (x_{m + 1}, \dots, x_{n}) . \frac{F_{x_{m + 1}}}{F_{x_{n}}} \dots γ (x_{m + 1}, \dots, x_{n}) \\ . \frac{F_{x_{n - 1}}}{F_{x_{n}}} γ (x_{m + 1}, \dots, x_{n})]^{T} . \end{matrix}

(4.9)

To establish equation , it suffices to argue that the vectors on the right-hand side are in the form of

\nabla G_{1}

and

\nabla G_{2}

for suitable functions

G_{1} (x_{1}, \dots, x_{m})

and

G_{2} (x_{m + 1}, \dots, x_{n})

⁠, to which the induction hypothesis can be applied by the same logic as before. Notice that the first one is dependent only on

x_{1}, \dots, x_{m}

⁠, while the second one is dependent only on

x_{m + 1}, \dots, x_{n}

⁠, again by equations and . For any

1 \leq i \leq m

and

m < j \leq n

⁠, we have

{(\frac{F_{x_{i}}}{F_{x_{1}}})}_{x_{j}} = 0

(respectively,

{(\frac{F_{x_{j}}}{F_{x_{n}}})}_{x_{i}} = 0

⁠) since there is no subtree of

T

that has only one of

x_{1}

and

x_{i}

(resp. only one of

x_{n}

and

x_{j}

⁠) and also

x_{j}

(resp. also

x_{i}

⁠). We finish the proof by verifying the corresponding integrability conditions,

{(β \frac{F_{x_{i}}}{F_{x_{1}}})}_{x_{i^{'}}} = {(β \frac{F_{x_{i^{'}}}}{F_{x_{1}}})}_{x_{i}}, {(γ \frac{F_{x_{j}}}{F_{x_{n}}})}_{x_{j^{'}}} = {(γ \frac{F_{x_{j^{'}}}}{F_{x_{n}}})}_{x_{j}},

for any

1 \leq i, i^{'} \leq m

and

m < j, j^{'} \leq n

⁠. In view of equation , one can change

β

and

γ

above to

\frac{F_{x_{1}}}{F_{x_{n}}}

or

\frac{F_{x_{n}}}{F_{x_{1}}}

⁠, respectively, and write the desired identities as the new ones,

\begin{matrix} {(\frac{F_{x_{1}}}{F_{x_{n}}} \frac{F_{x_{i}}}{F_{x_{1}}})}_{x_{i^{'}}} = {(\frac{F_{x_{1}}}{F_{x_{n}}} \frac{F_{x_{i^{'}}}}{F_{x_{1}}})}_{x_{i}}, {(\frac{F_{x_{n}}}{F_{x_{1}}} \frac{F_{x_{j}}}{F_{x_{n}}})}_{x_{j^{'}}} = {(\frac{F_{x_{n}}}{F_{x_{1}}} \frac{F_{x_{j^{'}}}}{F_{x_{n}}})}_{x_{j}}, \end{matrix}

which hold due to equation .

Remark 10.

As mentioned in remark 3, working with functions of the form 1.12 in theorem 7 rather than general smooth functions has the advantage of enabling us to determine a domain on which a superposition representation exists. In contrast, the sufficiency part of theorem 3 is a local statement since it relies on the implicit function theorem. It is possible to say something nontrivial about the domains when functions are furthermore analytic. This is because the implicit function theorem holds in the analytic category as well (Krantz & Parks, 2002, sec. 6.1) where lower bounds on the domain of validity of the theorem exist in the literature (Chang, He, & Prabhu, 2003).

5 Conclusion

In this article, we proposed a systematic method for studying smooth real-valued functions constructed as compositions of other smooth functions that are either of lower arity or in the form of a univariate activation function applied to a linear combination of inputs. We established that any such smooth superposition must satisfy nontrivial constraints in the form of algebraic PDEs, which are dependent only on the hierarchy of composition or, equivalently, only on the topology of the neural network that produces superpositions of this type. We conjectured that there always exist characteristic PDEs that also provide sufficient conditions for a generic smooth function to be expressible by the feedforward neural network in question. The genericity is to avoid singular cases and is captured by nonvanishing conditions that require certain polynomial functions of partial derivatives to be nonzero. We observed that there are also situations where nontrivial algebraic inequalities involving partial derivatives (PDIs) are imposed on the hierarchical functions. In summary, the conjecture aims to describe generic smooth functions computable by a neural network with finitely many universal conditions of the form $Φ \neq 0$ ⁠, $Ψ = 0$ ⁠, and $Θ > 0$ ⁠, where $Φ$ ⁠, $Ψ$ ⁠, and $Θ$ are polynomial expressions of the partial derivatives and are dependent only on the architecture of the network, not on any tunable parameter or any activation function used in the network. This is reminiscent of the notion of a semialgebraic set from real algebraic geometry. Indeed, in the case of compositions of polynomial functions or functions computed by polynomial neural networks, the PDE constraints yield equations for the corresponding functional variety in an ambient space of polynomials of a prescribed degree.

The conjecture was verified in several cases, most importantly, for tree architectures with distinct inputs where, in each regime, we explicitly exhibited a PDE characterization of functions computable by a tree network. Examples of tree architectures with repeated inputs were addressed as well. The proofs were mathematical in nature and relied on classical results of multivariable analysis.

The article moreover highlights the differences between the two regimes mentioned at the beginning: the hierarchical functions constructed out of composing functions of lower dimensionality and the hierarchical functions that are compositions of functions of the form $y \mapsto σ (〈 w, y 〉)$ ⁠. The former functions appear more often in the mathematical literature on the Kolmogorov-Arnold representation theorem, while the latter are ubiquitous in deep learning. The special form of functions $y \mapsto σ (〈 w, y 〉)$ requires more PDE constraints to be imposed on their compositions, whereas their mild nonlinearity is beneficial in terms of ascertaining the domain on which a claimed compositional representation exists.

Our approach for describing the functional spaces associated with feedforward neural networks is of natural interest in the study of expressivity of neural networks and could lead to new complexity measures. We believe that the point of view adapted here is novel and might shed light on a number of practical problems such as comparison of architectures and reverse-engineering deep networks.

Appendix A: Technical Proofs

Proof of Lemma 2.

We first prove that

F

can be extended to a coordinate system on the entirety of the box-like region

B

⁠, which we shall write as

I_{1} \times \dots \times I_{n}

⁠. As in the proof of theorem 3, we group the variables

x_{1}, \dots, x_{n}

according to the maximal subtrees of

T

in which they appear:

\begin{matrix} x_{1}, \dots, x_{m_{1}}; x_{m_{1} + 1}, \dots, x_{m_{1} + m_{2}}; \dots; x_{m_{1} + \dots + m_{l - 1} + 1}, \dots, x_{m_{1} + \dots + m_{l}}; \\ x_{m_{1} + \dots + m_{l} + 1}; \dots; x_{n}, \end{matrix}

where, denoting the subtrees emanating from the root of

T

by

T_{1}, \dots, T_{l}

⁠, for any

1 \leq s \leq l

the leaves of

T_{s}

are labeled by

x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s - 1} + m_{s}}

⁠; and

x_{m_{1} + \dots + m_{l} + 1}, \dots, x_{n}

represent the leaves that are directly connected to the root (if any); see Figure 13. Among the variables labeling the leaves of

T_{1}

⁠, there should exist one with respect to which the first-order partial derivative of

F

is not zero. Without any loss of generality, we may assume that

F_{x_{1}} \neq 0

at any point of

B

⁠. Hence, the Jacobian of the map

(F, x_{2}, \dots, x_{n}) : B \to R^{n}

is always invertible. To prove that the map provides a coordinate system, we just need to show that it is injective. Keeping

x_{2}, \dots, x_{n}

constant and varying

x_{1}

⁠, we obtain a univariate function of

x_{1}

on the interval

I_{1}

whose derivative is always nonzero and is hence injective.

Next, to prove that the level sets of

F : B \to R

are connected, notice that

F

admits a representation

\begin{matrix} F (x_{1}, \dots, x_{n}) = σ (w_{1} G_{1} + \dots + w_{l} G_{l} + w_{m_{1} + \dots + m_{l} + 1}^{'} x_{m_{1} + \dots + m_{l} + 1} + \dots + w_{n}^{'} x_{n}) \end{matrix}

(A.1)

where

G_{s} = G_{s} (x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s - 1} + m_{s}})

is the tree function that

T_{s}

computes by receiving

x_{m_{1} + \dots + m_{s - 1} + 1}, \dots, x_{m_{1} + \dots + m_{s - 1} + m_{s}}

from its leaves;

σ

is the activation function assigned to the root of

T

⁠; and

w_{1}, \dots, w_{l}

⁠,

w_{m_{1} + \dots + m_{l} + 1}^{'}, \dots, w_{n}^{'}

are the weights appearing at the root. A simple application of the chain rule implies that

G_{1}

⁠, which is a function implemented on the tree

T_{1}

⁠, satisfies the nonvanishing hypotheses of theorem 7 on the box-like region

I_{1} \times \dots \times I_{m_{1}}

⁠; moreover, the derivative of

σ

is nonzero at any point of its domain^⁹ because otherwise there exists a point of

p \in B

at which

F_{x_{i}} (p) = 0

for any leaf

x_{i}

⁠. By the same logic, the weight

w_{1}

must be nonzero because otherwise all first-order partial derivatives with respect to the variables appearing in

T_{1}

are identically zero. We now show that an arbitrary level set

L_{c} : = {x \in B | F (x) = c}

is connected. Given the representation (see equation ) of

F

⁠, the level set is empty if

σ

does not attain the value

c

⁠. Otherwise,

σ

attains

c

at a unique point

σ^{- 1} (c)

of its domain. So one may rewrite the equation

F (x_{1}, \dots, x_{n}) = c

as

\begin{matrix} G_{1} & = & - \frac{w_{2}}{w_{1}} G_{2} - \dots - \frac{w_{l}}{w_{1}} G_{l} - \frac{w_{m_{1} + \dots + m_{l} + 1}^{'}}{w_{1}} x_{m_{1} + \dots + m_{l} + 1} \\ - \dots - \frac{w_{n}^{'}}{w_{1}} x_{n} + \frac{1}{w_{1}} σ^{- 1} (c) . \end{matrix}

(A.2)

The left-hand side of equation is a function of

x_{1}, \dots, x_{m_{1}}

⁠, while its right-hand side, which we denote by

\tilde{G}

⁠, is a function of

x_{m_{1} + 1}, \dots, x_{n}

⁠. Therefore, the level set

L_{c}

is the preimage of

\{(y, \tilde{x}) \in R \times (I_{m_{1} + 1} \times \dots \times I_{n}) | y = \tilde{G} (\tilde{x})\}

(A.3)

under the map

\begin{matrix} \{\begin{matrix} π : B = (I_{1} \times \dots \times I_{m_{s}}) \times (I_{m_{1} + 1} \times \dots \times I_{n}) \to R \times (I_{m_{1} + 1} \times \dots \times I_{n}) \\ (x_{1}, \dots, x_{m_{1}}; \tilde{x}) \mapsto (G_{1} (x_{1}, \dots, x_{m_{1}}), \tilde{x}) . \end{matrix} \end{matrix}

(A.4)

The following simple fact can now be invoked: Let $π : X \to Y$ be a continuous map of topological spaces that takes open sets to open sets and has connected level sets. Then the preimage of any connected subset of $Y$ under $π$ is connected. Here,

L_{c}

is the preimage of the set from equation , which is connected since it is the graph of a continuous function, under the map

π

defined in equation , which is open because the scalar-valued function

G_{1}

is: its gradient never vanishes. Therefore, the connectedness of the level sets of

F

is implied by the connectedness of the level sets of

π

⁠. A level set of the map, equation , could be identified with a level set of its first component

G_{1}

⁠. Consequently, we have reduced to the similar problem for the function

G_{1}

⁠, which is implemented on the smaller tree

T_{1}

⁠. Therefore, an inductive argument yields the connectedness of the level sets of

F

⁠. It only remains to check the basic case of a tree whose leaves are directly connected to the root. In that setting,

F (x_{1}, \dots, x_{n})

is in the form of

σ (a_{1} x_{1} + \dots + a_{n} x_{n})

(the family of functions that lemma 3 is concerned with). By repeating the argument used before, the activation function

σ

is injective. Hence, a level set

F (x) = c

is the intersection of the hyperplane

a_{1} x_{1} + \dots + a_{n} x_{n} = σ^{- 1} (c)

with the box-like region

B

⁠. Such an intersection is convex and thus connected.

Proof of Lemma 3.

The necessity of conditions

F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}

follows from a simple computation. For the other direction, suppose

F_{x_{j}} \neq 0

throughout an open box-like region

B \subseteq R^{n}

and any ratio

\frac{F_{x_{i}}}{F_{x_{j}}}

is constant on

B

⁠. Denoting it by

a_{i}

⁠, we obtain numbers

a_{1}, \dots, a_{n}

with

a_{j} = 1

⁠. They form a vector

{[a_{1} \dots a_{n}]}^{T}

parallel to

\nabla F

⁠. Thus,

F

could have nonzero first-order partial derivative only with respect to the first member of the coordinate system,

(a_{1} x_{1} + \dots + a_{n} x_{n}, x_{1}, \dots, x_{j - 1}, x_{j + 1}, \dots, x_{n}),

for

B

⁠. The coordinate hypersurfaces are connected since they are intersections of hyperplanes in

R^{n}

with the convex region

B

⁠. This fact enables us to deduce that

F

can be written as a function of

a_{1} x_{1} + \dots + a_{n} x_{n}

globally.

Proof of Lemma 4.

For a function

q = q_{1} q_{2}

such as equation , equalities of the form

q q_{y_{a}^{(1)} y_{b}^{(2)}} = q_{y_{a}^{(1)}} q_{y_{b}^{(2)}}

hold since both sides coincide with

q_{1} q_{2} {(q_{1})}_{y_{a}^{(1)}} {(q_{2})}_{y_{b}^{(2)}}

⁠. For the other direction, let

q = q (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

be a smooth function on an open box-like region

B_{1} \times B_{2} \subseteq R^{n_{1}} \times R^{n_{2}}

that satisfies

q q_{y_{a}^{(1)} y_{b}^{(2)}} = q_{y_{a}^{(1)}} q_{y_{b}^{(2)}}

for any

1 \leq a \leq n_{1}

and

1 \leq b \leq n_{2}

⁠, and never vanishes. So

q

is either always positive or always negative. One may assume the former by replacing

q

with

- q

if necessary. Hence, we can define a new function

p : = Ln (q)

by taking the logarithm. We have

p_{y_{a}^{(1)} y_{b}^{(2)}} = {(\frac{q_{y_{a}^{(1)}}}{q})}_{y_{b}^{(2)}} = \frac{q q_{y_{a}^{(1)} y_{b}^{(2)}} - q_{y_{a}^{(1)}} q_{y_{b}^{(2)}}}{q^{2}} = 0 .

It suffices to show that this vanishing of mixed partial derivatives allows us to write

p (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

as

p_{1} (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}) + p_{2} (y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

since then exponentiating yields

q_{1}

and

q_{2}

as

e^{p_{1}}

and

e^{p_{2}}

⁠, respectively. The domain of

p

is a box-like region of the form

B_{1} \times B_{2} = (\prod_{a = 1}^{n_{1}} I_{a}^{(1)}) \times (\prod_{b = 1}^{n_{2}} I_{b}^{(2)}) .

Picking an arbitrary point

z_{1}^{(1)} \in I_{1}^{(1)}

⁠, the fundamental theorem of calculus implies

\begin{matrix} p (y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)}) \\ = & \int_{z_{1}^{(1)}}^{y_{1}^{(1)}} p_{y_{1}^{(1)}} (s_{1}^{(1)}, y_{2}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)}) d s_{1}^{(1)} \\ + p (z_{1}^{(1)}, y_{2}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)}) . \end{matrix}

On the right-hand side, the integral is dependent only on

y_{1}^{(1)}, \dots, y_{n_{1}}^{(1)}

because the partial derivatives of the integrand with respect to

y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)}

are all identically zero. The second term,

p (z_{1}^{(1)}, y_{2}^{(1)}, \dots, y_{n_{1}}^{(1)}; y_{1}^{(2)}, \dots, y_{n_{2}}^{(2)})

⁠, is a function on the smaller box-like region

(\prod_{a = 2}^{n_{1}} I_{a}^{(1)}) \times (\prod_{b = 1}^{n_{2}} I_{b}^{(2)})

in

R^{n_{1} - 1} \times R^{n_{2}}

and thus, proceeding inductively, can be brought into the appropriate summation form.

Appendix B: Differential Forms

Differential forms are ubiquitous objects in differential geometry and tensor calculus. We only need the theory of differential forms on open domains in Euclidean spaces. Theorem 5 (which has been used several times throughout the, for example, in the proof of theorem 3) is formulated in terms of differential forms. This appendix provides the necessary background for understanding the theorem and its proof.

We begin with a very brief account of the local theory of differential forms. (For a detailed treatment see Pugh, 2002, chap. 5.) Let

U

be an open subset of

R^{n}

⁠. A differential $k$ -form

ω

on

U

assigns a scalar to any

k

-tuple of tangent vectors at a point

p

of

U

⁠. This assignment, denoted by

ω_{p}

⁠, must be multilinear and alternating. We say

ω

is smooth (resp. analytic) if

ω_{p}

varies smoothly (resp. analytically) with

p

⁠. In other words, feeding

ω

with

k

smooth (resp. analytic) vector fields

V_{1}, \dots, V_{k}

on

U

results in a function

ω (V_{1}, \dots, V_{k}) : U \to R

that is smooth (resp. analytic). We next exhibit an expression for

ω

⁠. Consider the standard basis

(\frac{\partial}{\partial x_{1}}, \dots, \frac{\partial}{\partial x_{n}})

of vector fields on

U

where

\frac{\partial}{\partial x_{i}}

assigns

e_{i}

to each point. The dual basis is denoted by

(d x_{1}, \dots, d x_{n})

⁠, which at each point yields the dual of the standard basis

(e_{1}, \dots, e_{n})

for

R^{n}

⁠, that is,

d x_{i} (\frac{\partial}{\partial x_{j}}) \equiv δ_{i j}

⁠. Each of

d x_{1}, \dots, d x_{n}

is a 1-form on

U

⁠, and any

k

-form

ω

can be written in terms of them:

ω = \sum_{1 \leq i_{1} < \dots < i_{k} \leq n} f_{i_{1} \dots i_{k}} d x_{i_{1}} \land \dots \land d x_{i_{k}} .

(B.1)

Here, each coefficient

f_{i_{1} \dots i_{k}}

is a (smooth or analytic according to the context) function

U \to R

⁠. In front of it,

d x_{i_{1}} \land \dots \land d x_{i_{k}}

appears, which is a

k

-form satisfying

d x_{i_{1}} \land \dots \land d x_{i_{k}} (\frac{\partial}{\partial x_{i_{1}}}, \dots, \frac{\partial}{\partial x_{i_{k}}}) = 1 .

This is constructed by the operation of the exterior product (also called the wedge product) from multilinear algebra. The exterior product is an associative and distributive linear operation that out of

k_{i}

-tensors

τ_{i}

(1 \leq i \leq l)

constructs an alternating

(k_{1} + \dots + k_{l})

-tensor

τ_{1} \land \dots \land τ_{l}

⁠. This product is anti-commutative, for example,

d x_{i} \land d x_{j} = - d x_{j} \land d x_{i}

⁠; this is the reason that in equation , the indices are taken to be strictly ascending.

Another operation in the realm of differential forms is exterior differentiation. For the

k

-form

ω

from equation , its exterior derivative

d ω

is a

(k + 1)

-form defined as

d ω : = \sum_{1 \leq i_{1} < \dots < i_{k} \leq n} d f_{i_{1} \dots i_{k}} \land d x_{i_{1}} \land \dots \land d x_{i_{k}},

where the exterior derivative of a function

f

is defined as

d f : = \sum_{i = 1}^{n} f_{x_{i}} d x_{i} .

(B.2)

Notice that the 1-form is the dual of the gradient vector field

\nabla f = \sum_{i = 1}^{n} f_{x_{i}} \frac{\partial}{\partial x_{i}} .

(B.3)

B.1 Example 14

In dimension 3, the exterior differentiation encapsulates the familiar vector calculus operators curl and divergence. Consider the vector field

V (x, y, z) = {[V_{1} (x, y, z), V_{2} (x, y, z), V_{3} (x, y, z)]}^{T} .

The exterior derivatives

\begin{matrix} d (V_{1} d x + V_{2} d y + V_{3} d z) & = & ({(V_{3})}_{y} - {(V_{2})}_{z}) d y \land d z + ({(V_{1})}_{z} - {(V_{3})}_{x}) d z \land d x \\ + ({(V_{2})}_{x} - {(V_{1})}_{y}) d x \land d y \end{matrix}

and

\begin{matrix} d (V_{1} d y \land d z + V_{2} d z \land d x + V_{3} d x \land d y) \\ = ({(V_{1})}_{x} + {(V_{2})}_{y} + {(V_{3})}_{z}) d x \land d y \land d z, \end{matrix}

respectively, have

curl V

and

div V

as their coefficients. In fact, there is a general Stokes formula for differential forms that recovers the Kelvin–Stokes theorem and the divergence theorem as special cases. Finally, we point out that the familiar identities

curl \circ \nabla = 0

and

div \circ curl = 0

are instances of the general property

d \circ d = 0

of the exterior differentiation.

B.2 Example 15

As mentioned in the previous example, the outcome of twice applying the exterior differentiation operator to a form is always zero. This is an extremely important property that leads to the definitions of closed and exact differential forms. A $k$ -form $ω$ on an open subset $U$ of $R^{n}$ is called closed if $d ω = 0$ ⁠. This holds if $ω$ is in the form of $ω = d α$ for a $(k - 1)$ -form $α$ on $U$ ⁠. Such forms are called exact. The space of closed forms may be strictly larger than the space of exact forms; the difference of these spaces can be used to measure the topological complexity of $U$ ⁠. If $U$ is an open box-like region, every closed form on it is exact. But, for instance, the 1-form $ω = - \frac{y}{x^{2} + y^{2}} d x + \frac{x}{x^{2} + y^{2}} d y$ on $R^{2} - {(0, 0)}$ is closed while it may not be written as $d α$ for any smooth function $α : R^{2} - {(0, 0)} \to R$ ⁠. This brings us to a famous fact from multivariable calculus that we have used several times (e.g., in the proof of theorem 4). A necessary condition for a vector field $V = \sum_{i = 1}^{n} V_{i} \frac{\partial}{\partial x_{i}}$ on an open subset $U$ of $R^{n}$ to be a gradient vector field is ${(V_{i})}_{x_{j}} = {(V_{j})}_{x_{i}}$ for any $1 \leq i, j \leq n$ ⁠. Near each point of $U$ ⁠, the vector field $V$ may be written as $\nabla f$ ⁠; it is globally in the form of $\nabla f$ for a function $f : U \to R$ when $U$ is simply connected. In view of equations B.2 and B.3, one may rephrase this fact as: Closed 1-forms on $U$ are exact if and only if $U$ is simply connected.

Proof of Theorem 5.

Near a point $p \in R^{n}$ at which $V (p) \neq 0$ ⁠, we seek a locally defined function $ξ$ with $V ∥ \nabla ξ \neq 0$ ⁠. Recall that if $q \in R^{n}$ is a regular point of $ξ$ ⁠, then near $q$ ⁠, the level set of $ξ$ passing through $q$ is an $(n - 1)$ -dimensional submanifold of $R^{n}$ to which the gradient vector field, $\nabla ξ \neq 0$ ⁠, is perpendicular. As we want the gradient to be parallel to the vector field $V$ ⁠, the equivalent characterization in terms of the 1-form $ω$ ⁠, which is the dual of $V$ (cf. equations 3.1 and 3.3), asserts that $ω$ is zero at any vector tangent to the level set. So the tangent space to the level set at the point $q$ could be described as ${v \in R^{n} | ω_{q} (v) = 0}$ ⁠. As $q$ varies near $p$ ⁠, these $(n - 1)$ -dimensional subspaces of $R^{n}$ vary smoothly. In differential geometry, such a higher-dimensional version of a vector field is called a distribution, and the property that these subspaces are locally given by tangent spaces to a family of submanifolds (the level sets here) is called integrability. The seminal Frobenius theorem (Narasimhan, 1968, theorem 2.11.11) implies that the distribution defined by a nowhere vanishing 1-from $ω$ is integrable if and only if $ω \land d ω = 0$ ⁠.

Notes

¹

To be mathematically precise, the open neighborhood of $p$ on which $F$ admits a compositional representation in the desired form may be dependent on $F$ and $p$ ⁠. So conjecture 1 is local in nature and must be understood as a statement about function germs.

²

Convergence in the $C^{r}$ -norm is defined as the uniform convergence of the function and its partial derivatives up to order $r$ ⁠.

³

In conjecture 1, the subset cut off by equations $Ψ_{i} = 0$ is meager: It is a closed and (due to the term nonvacuous appearing in the conjecture) proper subset of the space of functions computable by $N$ ⁠, and a function implemented by $N$ at which a $Ψ_{i}$ vanishes could be perturbed to another computable function at which all of $Ψ_{i}$ 's are nonzero.

⁴

An open box-like region in $R^{n}$ is a product $I_{1} \times \dots \times I_{n}$ of open intervals.

⁵

A piece of terminology introduced in Farhoodi et al. (2019) may be illuminating here. A member of a triple $(x_{i}, x_{j}, x_{k})$ of (not necessarily distinct) leaves of $T$ is called the outsider of the triple if there is a (rooted full) subtree of $T$ that misses it but has the other two members. Theorem 3 imposes $F_{x_{i} x_{k}} F_{x_{j}} = F_{x_{j} x_{k}} F_{x_{i}}$ whenever $x_{k}$ is the outsider, while theorem 4 imposes the constraint whenever $x_{i}$ and $x_{j}$ are not outsiders.

⁶

Notice that this is the best one can hope to recover because through scaling the weights and inversely scaling the inputs of activation functions, the function $F$ could also be written as $σ (\tilde{τ} (λ a x + λ b y) + c z + d w)$ or $σ (\tilde{τ_{1}} (λ a x + λ b y) + τ_{2} (c z + d w))$ where $\tilde{τ} (y) : = τ (\frac{y}{λ})$ and $\tilde{τ_{1}} (y) : = τ_{1} (\frac{y}{λ})$ ⁠. Thus, the other ratios $\frac{a}{c}$ and $\frac{b}{d}$ are completely arbitrary.

⁷

Compare with the proof of lemma 4 in appendix A.

⁸

A single vertex is not considered to be a rooted tree in our convention.

⁹

As the vector $(x_{1}, \dots, x_{n})$ of inputs varies in the box-like region $B$ ⁠, the inputs to each node form an interval on which the corresponding activation function is defined.

References

Arnold

,

V. I.

(

2009a

).

On the representation of functions of several variables as a superposition of functions of a smaller number of variables.

In

V. A.

Arnold

,

Collected works: Representations of functions, celestial mechanics and KAM theory, 1957–1965

(pp.

25

–

46

).

Berlin

:

Springer

.

Google Scholar

Arnold

,

V. I.

(

2009b

).

Representation of continuous functions of three variables by the superposition of continuous functions of two variables.

In

V. A.

Arnold

,

Collected works: Representations of functions, celestial mechanics and KAM theory, 1957–1965

(pp.

47

–

133

).

Berlin

:

Springer

.

Google Scholar

Bartlett

,

P. L.

,

Maiorov

,

V.

, &

Meir

,

R.

(

1999

). Almost linear VC dimension bounds for piecewise polynomial networks. In

S.

Solla

,

T.

Leen

, &

K. R.

Müller

(Eds.),

Advances in neural information processing systems, 11

(pp.

190

–

196

).

Cambridge, MA

:

MIT Press

.

Google Scholar

Bianchini

,

M.

, &

Scarselli

,

F.

(

2014

).

On the complexity of neural network classifiers: A comparison between shallow and deep architectures

.

IEEE Transactions on Neural Networks and Learning Systems

,

25

(

8

),

1553

–

1565

.

https://doi.org/10.1109/TNNLS.2013.2293637

[PubMed]

Google Scholar

Crossref

PubMed

Boyce

,

W. E.

, &

DiPrima

,

R. C.

(

2012

).

Elementary differential equations

(10th ed.).

Hoboken, NJ

:

Wiley

.

Google Scholar

Brattka

,

V.

(

2007

). From Hilbert's 13th problem to the theory of neural networks: Constructive aspects of Kolmogorov's superposition theorem. In

E.

Charpentier

,

A.

Lesne

, &

N.

Annick

(Eds.),

Kolmogorov's heritage in mathematics

(pp.

253

–

280

).

Berlin

:

Springer

.

Google Scholar

Crossref

Buck

,

R. C.

(

1976

).

Approximate complexity and functional representation

(Tech. Rep.).

Madison

:

Mathematics Research Center, University of Wisconsin, Madison.

https://apps.dtic.mil/dtic/tr/fulltext/u2/a031972.pdf

Google Scholar

Buck

,

R. C.

(

1979

).

Approximate complexity and functional representation

.

J. Math. Anal. Appl.

,

70

(

1

),

280

–

298

.

https://doi.org/10.1016/0022-247X(79)90091-X

Google Scholar

Crossref

Buck

,

R. C.

(

1981a

).

Characterization of classes of functions.

Amer. Math. Monthly

,

88

(

2

),

139

–

142

.

https://doi.org/10.1080/00029890.1981.11995204

Google Scholar

Crossref

Buck

,

R. C.

(

1981b

).

The solutions to a smooth PDE can be dense in

C

(I).

J. Differential Equations

,

41

(

2

),

239

–

244

.

https://doi.org/10.1016/0022-0396(81)90060-7

Google Scholar

Crossref

Chang

,

H.-C.

,

He

,

W.

, &

Prabhu

,

N.

(

2003

).

The analytic domain in the implicit function theorem.

J. Inequal. Pure Appl. Math.

,

4

(

1

), art.

[PubMed]

.

Google Scholar

Chatziafratis

,

V.

,

Nagarajan

,

S. G.

,

Panageas

,

I.

, &

Wang

,

X.

(

2019

).

Depth-width trade-offs for ReLU networks via Sharkovsky's theorem

. arXiv:1912.04378v1.

Google Scholar

Cohen

,

N.

,

Sharir

,

O.

, &

Shashua

,

A.

(

2016

).

On the expressive power of deep learning: A tensor analysis.

In

Proceedings of the Conference on Learning Theory

(pp.

698

–

728

).

Google Scholar

Coste

,

M.

(

2000

).

An introduction to semialgebraic geometry

. Citeseer.

Google Scholar

Cybenko

,

G.

(

1989

).

Approximation by superpositions of a sigmoidal function

.

Mathematics of Control, Signals and Systems

,

2

(

4

),

303

–

314

.

https://doi.org/10.1007/BF02551274

Google Scholar

Crossref

Dehmamy

,

N.

,

Rohani

,

N.

, &

Katsaggelos

,

A. K.

(

2019

).

Direct estimation of weights and efficient training of deep neural networks without SGD

. In

Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing

(pp.

3232

–

3236

).

Piscataway, NJ

:

IEEE

.

Google Scholar

Du

,

S. S.

, &

Lee

,

J. D.

(

2018

).

On the power of over-parametrization in neural networks with quadratic activation

. arXiv:1803.01206v2.

Google Scholar

Eldan

,

R.

, &

Shamir

,

O.

(

2016

).

The power of depth for feedforward neural networks.

In

Proceedings of the Conference on Learning Theory

(pp.

907

–

940

).

Google Scholar

Farhoodi

,

R.

,

Filom

,

K.

,

Jones

,

I. S.

, &

Körding

,

K. P.

(

2019

).

On functions computed on trees.

Neural Computation

,

31

,

2075

–

2137

.

https://doi.org/10.1162/neco_a_01231

[PubMed]

Google Scholar

Crossref

PubMed

Fefferman

,

C.

, &

Markel

,

S.

(

1994

). Recovering a feed-forward net from its output. In

G.

Tesauro

,

D.

Toretsky

, &

T.

Leen

(Eds.),

Advances in neural information processing systems

,

7

(pp.

335

–

342

).

Cambridge, MA

:

MIT Press

.

Google Scholar

Gerhard

,

S.

,

Andrade

,

I.

,

Fetter

,

R. D.

,

Cardona

,

A.

, &

Schneider-Mizell

,

C. M.

(

2017

).

Conserved neural circuit structure across drosophila larval development revealed by comparative connectomics

.

eLife

,

6

, e29089.

https://doi.org/10.7554/eLife.29089

[PubMed]

Google Scholar

Gillette

,

T. A.

, &

Ascoli

,

G. A.

(

2015

).

Topological characterization of neuronal arbor morphology via sequence representation: I-motif analysis

.

BMC Bioinformatics

,

16

(

1

), 216.

Google Scholar

Girosi

,

F.

, &

Poggio

,

T.

(

1989

).

Representation properties of networks: Kolmogorov's theorem is irrelevant

.

Neural Computation

,

1

(

4

),

465

–

469

.

https://doi.org/10.1162/neco.1989.1.4.465

Google Scholar

Crossref

Hecht-Nielsen

,

R.

(

1987

).

Kolmogorov's mapping neural network existence theorem.

In

Proceedings of the International Conference on Neural Networks

(vol.

3

, pp.

11

–

14

).

Piscataway, NJ

:

IEEE

.

Google Scholar

Hilbert

,

D.

(

1902

).

Mathematical problems

.

Bulletin of the American Mathematical Society

,

8

(

10

),

437

–

479

.

https://doi.org/10.1090/S0002-9904-1902-00923-3

Google Scholar

Crossref

Hornik

,

K.

(

1991

).

Approximation capabilities of multilayer feedforward networks

.

Neural Networks

,

4

(

2

),

251

–

257

.

https://doi.org/10.1016/0893-6080(91)90009-T

Google Scholar

Crossref

Hornik

,

K.

,

Stinchcombe

,

M.

, & White H. (

1989

).

Multilayer feedforward networks are universal approximators

.

Neural Networks

,

2

(

5

),

359

–

366

.

https://doi.org/10.1016/0893-6080(89)90020-8

Google Scholar

Crossref

Kileel

,

J.

,

Trager

,

M.

, &

Bruna

,

J.

(

2019

).

On the expressive power of deep polynomial neural networks

. arXiv:1905.12207v1.

Google Scholar

Kollins

,

K. M.

, &

Davenport

,

R. W.

(

2005

). Branching morphogenesis in vertebrate neurons. In

K.

Kollins

&

R.

Davenport

(Eds.),

Branching morphogenesis

(pp.

8

–

65

).

Berlin

:

Springer

.

Google Scholar

Crossref

Kolmogorov

,

A. N.

(

1957

).

On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition

.

Dokl. Akad. Nauk SSSR

,

114

,

953

–

956

.

Google Scholar

Krantz

,

S. G.

, &

Parks

,

H. R.

(

2002

).

The implicit function theorem

.

Boston

:

Birkhäuser

.

Google Scholar

Kůrková

,

V.

(

1991

).

Kolmogorov's theorem is relevant

.

Neural Computation

,

3

(

4

),

617

–

622

.

Google Scholar

Crossref

PubMed

Kůrková

,

V.

(

1992

).

Kolmogorov's theorem and multilayer neural networks

.

Neural Networks

,

5

(

3

),

501

–

506

.

Google Scholar

Crossref

Lin

,

H. W.

,

Tegmark

,

M.

, &

Rolnick

,

D.

(

2017

).

Why does deep and cheap learning work so well?

Journal of Statistical Physics

,

168

(

6

),

1223

–

1247

.

https://doi.org/10.1007/s10955-017-1836-5

Google Scholar

Crossref

Lorentz

,

G. G.

(

1966

).

Approximation of functions

.

New York

:

Holt

.

Google Scholar

Mhaskar

,

H.

(

1996

).

Neural networks for optimal approximation of smooth and analytic functions

.

Neural Computation

,

8

(

1

),

164

–

177

.

https://doi.org/10.1162/neco.1996.8.1.164

Google Scholar

Crossref

Mhaskar

,

H.

,

Liao

,

Q.

, &

Poggio

,

T.

(

2017

).

When and why are deep networks better than shallow ones?

In

Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.

Palo Alto

:

AAAI

.

Google Scholar

Minsky

,

M.

, &

Papert

,

S. A.

(

2017

).

Perceptrons: An introduction to computational geometry

.

Cambridge, MA: MIT Press

.

Google Scholar

Montufar

,

G. F.

,

Pascanu

,

R.

,

Cho

,

K.

, &

Bengio

,

Y.

(

2014

). On the number of linear regions of deep neural networks. In

Z.

Ghahramani

,

M.

Welling

,

C.

Cortes

,

N.

Lawrence

, &

K. Q.

Weinberger

(Eds.),

Advances in neural information processing systems, 27

(pp.

2924

–

2932

).

Red Hook, NY

:

Curran

.

Google Scholar

Narasimhan

,

R.

(

1968

).

Analysis on real and complex manifolds

.

Amsterdam

:

North-Holland

.

Google Scholar

Ostrowski

,

A.

(

1920

).

Über dirichletsche reihen und algebraische differentialgleichungen

.

Mathematische Zeitschrift

,

8

(

3

),

241

–

298

.

https://doi.org/10.1007/BF01206530

Google Scholar

Crossref

Petersen

,

P.

,

Raslan

,

M.

, &

Voigtlaender

,

F.

(

2020

).

Topological properties of the set of functions generated by neural networks of fixed size

.

Foundations of Computational Mathematics

,

21

,

375

–

444

.

https://doi.org/10.1007/s10208-020-09461-0

Google Scholar

Crossref

Poggio

,

T.

,

Banburski

,

A.

, &

Liao

,

Q.

(

2019

).

Theoretical issues in deep networks: Approximation, optimization and generalization.

arXiv:1908.09375v1.

Google Scholar

Poggio

,

T.

,

Mhaskar

,

H.

,

Rosasco

,

L.

,

Miranda

,

B.

, &

Liao

,

Q.

(

2017

).

Why and when can deep—but not shallow—networks avoid the curse of dimensionality: A review

.

International Journal of Automation and Computing

,

14

(

5

),

503

–

519

.

https://doi.org/10.1007/s11633-017-1054-2

Google Scholar

Crossref

Pólya

,

G.

, &

Szegö

,

G.

(

1945

).

Aufgaben und Lehrsätze aus der Analysis

.

New York

:

Dover

.

Google Scholar

Poole

,

B.

,

Lahiri

,

S.

,

Raghu

,

M.

,

Sohl-Dickstein

,

J.

, &

Ganguli

,

S.

(

2016

). Exponential expressivity in deep neural networks through transient chaos. In

D.

Lee

,

M.

Sugiyama

,

U.

Luxburg

,

I.

Guyon

, &

R.

Garnett

(Eds.),

Advances in neural information processing systems

,

29

(pp.

3360

–

3368

).

Red Hook, NY

:

Curran

.

Google Scholar

Pugh

,

C. C.

(

2002

).

Real mathematical analysis

.

Springer-Verlag

,

New York

.

Google Scholar

Raghu

,

M.

,

Poole

,

B.

,

Kleinberg

,

J.

,

Ganguli

,

S.

, &

Dickstein

,

J. S.

(

2017

).

On the expressive power of deep neural networks

. In

Proceedings of the 34th International Conference on Machine Learning

,

70

(pp.

2847

–

2854

).

Google Scholar

Rolnick

,

D.

, &

Kording

,

K. P.

(

2019, October

).

Reverse-engineering deep ReLU networks.

arXiv:1910.00744v2.

Google Scholar

Rubel

,

L. A.

(

1981

).

A universal differential equation

.

Bull. Amer. Math. Soc. (N.S.)

,

4

(

3

),

345

–

349

.

https://doi.org/10.1090/S0273-0979-1981-14910-7

Google Scholar

Crossref

Schneider-Mizell

,

C. M.

,

Gerhard

,

S.

,

Longair

,

M.

,

Kazimiers

,

T.

,

Li

,

F.

,

Zwart

,

M. F.

, …

Cardona

,

A.

(

2016

).

Quantitative neuroanatomy for connectomics in drosophila

.

eLife

,

5

, e12059.

https://doi.org/10.7554/eLife.12059

Google Scholar

Soltanolkotabi

,

M.

,

Javanmard

,

A.

, &

Lee

,

J. D.

(

2018

).

Theoretical insights into the optimization landscape of over-parameterized shallow neural networks

.

IEEE Transactions on Information Theory

,

65

(

2

),

742

–

769

.

https://doi.org/10.1109/TIT.2018.2854560

Google Scholar

Crossref

Sprecher

,

D. A.

(

1965

).

On the structure of continuous functions of several variables

.

Trans. Amer. Math. Soc.

,

115

,

340

–

355

.

https://doi.org/10.2307/1994273

Google Scholar

Crossref

Telgarsky

,

M.

(

2016

).

Benefits of depth in neural networks.

arXiv:1602.04485v2.

Google Scholar

Venturi

,

L.

,

Bandeira

,

A. S.

, &

Bruna

,

J.

(

2018

).

Spurious valleys in two-layer neural network optimization landscapes.

arXiv:1802.06384v3.

Google Scholar

Vituškin

,

A. G.

(

1954

).

On Hilbert's thirteenth problem

.

Doklady Akad. Nauk SSSR (N.S.)

,

95

,

701

–

704

.

Google Scholar

Vituškin

,

A. G.

(

1964

).

A proof of the existence of analytic functions of several variables not representable by linear superpositions of continuously differentiable functions of fewer variables

.

Dokl. Akad. Nauk SSSR

,

156

,

1258

–

1261

.

Google Scholar

Vituškin

,

A. G.

(

2004

).

On Hilbert's thirteenth problem and related questions

.

Russian Mathematical Surveys

,

59

(

1

), 11.

Google Scholar

Vituškin

,

A. G.

, &

Henkin

,

G. M.

(

1967

).

Linear superpositions of functions

.

Uspehi Mat. Nauk

,

22

(

133

),

77

–

124

.

Google Scholar

von Golitschek

,

M.

(

1980

).

Remarks on functional representation.

In

Approximation theory, III (Proc. Conf., Univ. Texas, Austin, Tex., 1980)

(pp.

429

–

434

).

New York

:

Academic Press

.

Google Scholar

2021

Massachusetts Institute of Technology

On PDE Characterization of Smooth Hierarchical Functions Computed by Neural Networks

Abstract

1 Introduction

1.1 Motivation

1.1.1 Example 1

1.1.2 Example 2

1.2 Statements of Main Results

1.2.1 Example 3

1.3 Related Work

1.4 Outline of the Article

2 Existence of PDE Constraints

3 Toy Examples

3.1 Trees with Four Inputs

3.1.1 Example 4

3.1.2 Example 5

3.1.3 Example 6

3.1.4 Example 7

3.1.5 Example 8

3.2 Examples of Functions Computed by Neural Networks

3.2.1 Example 9

3.2.2 Example 10

3.2.3 Example 11

3.3 Examples of Polynomial Neural Networks

3.3.1 Example 12

3.3.2 Example 13

4 PDE Characterization of Tree Functions

5 Conclusion

Appendix A: Technical Proofs

Appendix B: Differential Forms

B.1 Example 14

B.2 Example 15

Notes

References

Email alerts

Cited By

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

On PDE Characterization of Smooth Hierarchical Functions Computed by Neural Networks

Abstract

1 Introduction

1.1 Motivation

1.1.1 Example 1

1.1.2 Example 2

1.2 Statements of Main Results

1.2.1 Example 3

1.3 Related Work

1.4 Outline of the Article

2 Existence of PDE Constraints

3 Toy Examples

3.1 Trees with Four Inputs

3.1.1 Example 4

3.1.2 Example 5

3.1.3 Example 6

3.1.4 Example 7

3.1.5 Example 8

3.2 Examples of Functions Computed by Neural Networks

3.2.1 Example 9

3.2.2 Example 10

3.2.3 Example 11

3.3 Examples of Polynomial Neural Networks

3.3.1 Example 12

3.3.2 Example 13

4 PDE Characterization of Tree Functions

5 Conclusion

Appendix A: Technical Proofs

Appendix B: Differential Forms

B.1 Example 14

B.2 Example 15

Notes

References

Email alerts

Cited By

Related Articles

Related Book Chapters

A product of The MIT Press

MIT Press Direct

Information

MIT Press

Contact Us

This Feature Is Available To Subscribers Only