Trade-Offs Between Energy and Depth of Neural Networks

Nervous systems receive an abundance of environmental stimuli and encode them within neural networks, in which the internal representations formed play a crucial role in fine neural information processing. More formally, DiCarlo and Cox (2007) considered an internal representation as a firing pattern, that is, a vector in a very high-dimensional space, where each axis is one neuron’s activity, and the dimensionality equals the number (e.g., approximately 1 million) of neurons in a feedforward neural network and then argued that for object recognition tasks, a representation is considered to be good if, for a given pair of images that are hard to distinguish in the input space, there exist representations that are easy to separate by simple classifiers such as linear classifiers.

However, there is a severe constraint toward forming useful internal representations. While the total energy resources supplied to the brain are limited, there are high energetic costs incurred by both the resting and signaling of neurons (Attwell & Laughlin, 2001; Lennie, 2003). Niven and Laughlin (2008) claimed that selective pressures both to improve behavioral performance and reduce energy consumption can affect all levels of components and mechanisms in the brain. Thus, in addition to the development of subcellular structures and individual neurons, neural networks are likely to develop useful and energy-efficient internal representations.

Sparse coding is a strategy in which a relatively small number of neurons are simultaneously active out of a large population and is considered a plausible principle for constructing internal representations in the brain. Sparse coding can reconcile the issue of representational capacity and energy expenditure (Földiák, 2003; Levy & Baxter, 1996; Olshausen & Field, 2004) and has been experimentally observed in various sensory systems (see, for example, Barth & Poulet, 2012; Shoham et al., 2006), including the visual cortex, where a representation of a few active neurons conveys information useful enough to reconstruct or classify natural images (Tang et al., 2018; Yoshida & Ohki, 2020).

Because sparse coding restricts the available internal representations of a neural network, it limits its representational power. Can we quantitatively evaluate the effects of a sparse coding strategy on neural information processing?

Uchizawa et al. (2008) sought to address the question from the viewpoint of circuit complexity. Let $C$ be a class of feedforward circuits modeling neural networks. In a typical circuit complexity argument, we introduce a complexity measure for Boolean functions and show that any Boolean function computable by a circuit $C∈C$ has a low complexity bounded by its available computational resources for $C$⁠. If a Boolean function $f$ has inherently high complexity regarding its scale, we can conclude that any circuit in $C$ requires a sufficiently large number of resources to compute $f$⁠. A Boolean function can model a binary classification task, implying that neural networks cannot construct good internal representations for the task if the computational resources are limited or, equivalently, the scale of the task is too large.

More formally, Uchizawa et al. (2008) employed threshold circuits as a model for neural networks, where a threshold circuit is a feedforward logic circuit whose basic computational element computes a linear threshold function (McCulloch & Pitts, 1943; Minsky & Papert, 1988; Parberry, 1994; Rosenblatt, 1958; Siu & Bruck, 1991; Siu et al., 1995; Siu & Roychowdhury, 1994). Size, depth, and weight are computational resources that have been extensively studied. For a threshold circuit $C$⁠, size $s$ is defined as the number of gates in $C$⁠, depth $d$ is the number of layers of $C$⁠, and weight $w$ is the degree of resolution of the weights among the gates. These resources are likely to be bound by neural networks in the brain; the number of neurons in the brain is clearly limited, and the number of neurons performing a particular task could be further restricted. Depth is related to the reaction time required for performing tasks; therefore, low depth values are preferred. A single synaptic weight may take an analog value; however, it is unlikely that a neuron will have an infinitely high resolution against neuronal noise; hence, a neuron may have a bounded degree of resolution. In addition, inspired by the sparse coding, Uchizawa et al. (2008) introduced a new complexity measure called energy complexity, where the energy of a circuit is defined as the maximum number of internal gates outputting nonzero values taken over all the input assignments to the circuit. Studies on the energy complexities of other types of logic circuits have also been reported: Dinesh et al. (2020), Kasim-zade (1992), Silva and Souza (2022), and Sun et al. (2019); Vaintsvaig (1961).

Uchizawa et al. (2008) then showed that the energy-bounded threshold circuits have a certain computational power by observing that threshold circuits can simulate linear decision trees, where a linear decision tree is a binary decision tree in which a query at each internal node is given by a linear threshold function. In particular, they proved that any linear decision tree of $ℓ$ leaves can be simulated by a threshold circuit of size $O(ℓ)$ and energy $O(logℓ)$⁠. Hence, any linear decision tree of $poly (n)$ leaves can be simulated by a threshold circuit of size $poly (n)$ and energy only $O(logn)$⁠, where $n$ is the number of input variables.

Following Uchizawa et al. (2008), a sequence of papers shows relations among other resources such as size, depth, and weight (Maniwa et al., 2018; Suzuki et al., 2011, 2013; Uchizawa & Takimoto, 2008; Uchizawa, 2020, 2014; Uchizawa et al., 2011). In particular, Uchizawa and Takimoto (2008) showed that any threshold circuit $C$ of depth $d$ and energy $e$ requires size $s=2Ω(n/ed)$ if $C$ computes a high bounded-error communication complexity function such as the inner product modulo 2. Even for low communication complexity functions, an exponential lower bound on the size is known for constant-depth threshold circuits: any threshold circuit $C$ of depth $d$ and energy $e$ requires size $s=2Ω(n/e2e+dlogen)$ if $C$ computes the parity function (Uchizawa, 2020). These results provide exponential lower bounds if the depth is constant and energy is sublinear (Uchizawa & Takimoto, 2008) or sublogarithmic (Uchizawa, 2020), while both the inner product modulo 2 and parity are computable by linear-size, constant-depth, and linear-energy threshold circuits. These results imply that the energy is strongly related to the representational power of threshold circuits and is an important computational resource. However, these lower bounds break down when we consider threshold circuits of larger depth and energy, say, nonconstant depth and sublinear energy.

Here, we provide a more sophisticated relation among size, depth, energy, and weight. Our main result is formulated as follows. Let $f$ be a Boolean function of $n$ variables and $(X,Y)$ be a binary partition of ${1,2,...,n}$⁠. Then we can express any assignment $ξ∈{0,1}n$ as $(a,b)∈{0,1}|X|×{0,1}|Y|$⁠. A communication matrix $MfX:Y$ is a $2|X|×2|Y|$ matrix, where each row (resp., each column) is indexed by an assignment $a∈{0,1}|X|$ (resp., $b∈{0,1}|Y|$⁠), and the value $MfX:Y[a,b]$ is defined to be the output of $f$ given $a$ and $b$⁠. We denote by $rk (f)$ the maximum rank of $MfX:Y$ over real numbers, where the maximum is to take over all the partitions $(X,Y)$ of ${1,2,...,n}$⁠. We establish the following relation.

The theorem also improves known lower bounds for threshold circuits. By rearranging equation 1.2, the following lower bound can be obtained: $2n/(2ed)/(wn)≤s$⁠, which is exponential in $n$ if both $d$ and $e$ are sublinear and $w$ is subexponential. For example, an exponential lower bound $s=2Ω(n1/3)$ can be obtained even for threshold circuits of depth $n1/3$⁠, energy $n1/3$⁠, and weight $2o(n1/3)$⁠. Similar lower bounds can be obtained for the inner product modulo 2 and the equality function because they have linear ranks. Upon comparing the lower bound $s=2Ω(n/ed)$ from the study by Uchizawa and Takimoto (2008) to ours, our lower bound is found to be meaningful only for subexponential weight but affords two-fold improvement: the lower bound is exponential even if $d$ is sublinear, and it provides an exponential lower bound for Boolean functions with a much weaker condition such that Boolean functions has rank $Ω(n)$ over real numbers. Threshold circuits have received considerable attention in circuit complexity, and several lower-bound arguments have developed for threshold circuits under some restrictions on computational resources including size, depth, energy, and weight (Amano, 2020; Amano & Maruoka, 2005; Chen et al., 2018; Hajnal et al., 1993; Håstad & Goldmann, 1991; Impagliazzo et al., 1997; Kane & Williams, 2016; Nisan, 1993; Razborov & Sherstov, 2010; Sherstov, 2007; Uchizawa, 2020; Uchizawa & Takimoto, 2008; Uchizawa et al., 2011). However, the arguments for lower bounds are designated for constant-depth threshold circuits and cannot provide meaningful bounds when the depth is not constant. In particular, $CD n$ and $EQ n$ are computable by polynomial-size and constant-depth threshold circuits. Thus, directly applying known techniques is unlikely to yield our lower bound.

To complement theorem 1, we show that the relation is tight up to a constant factor if the products of $e$ and $d$ are small.

Apart from threshold circuits, we consider other well-studied models of neural networks, where an activation function and weights are discretized (e.g., discretized sigmoid and ReLU circuits). The size, depth, energy, and weight are also important parameters for artificial neural networks. The size and depth are major topics on the success of deep learning. The energy is related to important techniques for deep learning methods such as sparse autoencoder (Ng, 2011). The weight resolution is closely related to chip resources in neuromorphic hardware systems (Pfeil et al., 2012); accordingly, quantization schemes have received considerable attention (Courbariaux et al., 2015; Hubara et al., 2018).

For discretized circuits, we also show that there exists a similar trade-off to the one for threshold circuits. For example, the following proposition can be obtained for discretized sigmoid circuits:

Therefore, artificial neural networks obtained through the machine learning process may face a challenge to obtain good internal representations; because hardware constraints are imposed on the number of neurons and the weight resolution, the depth is predefined, and the learning algorithm may force a resulting network to acquire sparse activity. Further, $c$ times larger depth is comparable to $2c$ times larger size. Thus, increasing the depth could significantly help neural networks to increase their expressive power and can evidently aid neural networks in acquiring sparse activity. Consequently, our bound may afford insight into the reason for the success of deep learning.

The remainder of the article is organized as follows. In section 2, we formally introduce the terminologies and notations that are used in the rest of the article. In section 3, we present our main lower-bound result. In section 4, we show the tightness of the lower bound. In section 5, we show a bound for discretized circuits. In section 6, we conclude with some remarks.

For an integer $n$⁠, we denote by $[n]$ a set ${1,2,...n}$⁠. For a finite set $Z$ of integers, we denote by ${0,1}Z$ a set of $2|Z|$ Boolean assignments, where each assignment consists of $|Z|$ elements, each of which is indexed by $i∈Z$⁠. The base of the logarithm is two unless stated otherwise. In section 2.1, we define terms on threshold circuits and discretized circuits. In section 2.2, we define the communication matrix together with some related terms and summarize some facts.

A threshold circuit $C$ is a combinatorial circuit consisting of threshold gates and is expressed by a directed acyclic graph. The nodes of in-degree 0 correspond to input variables, and the other nodes correspond to gates. Let $G$ be a set of the gates in $C$⁠. For each gate $g∈G$⁠, the level of $g$⁠, denoted by $lev (g)$⁠, is defined as the length of a longest path from an input variable to $g$ on the underlying graph of $C$⁠. For each $ℓ∈[d]$⁠, we define $Gℓ$ as a set of gates in the $ℓ$th level: $Gℓ={g∈G∣ lev (g)=ℓ}$⁠. We denote by $g clf$ a unique output gate, which is a linear classifier separating internal representations given by the gates in the lower levels (possibly together with input variables). We say that $C$computes a Boolean function $f:{0,1}[n]→{0,1}$ if $g clf (ξ)=f(ξ)$ for every $ξ∈{0,1}[n]$⁠. Although the inputs to a gate $g$ in $C$ may not be only the input variables but the outputs of gates in the lower levels, we write $g(ξ)$ for the output of $g$ for $ξ∈{0,1}[n]$ because $ξ$ inductively decides the output of $g$⁠.

Let $ϕ$ be an activation function. Let $δ$ be a discretizer that maps a real number to a number representable by a bit width $b$⁠. We define a discretized activation function $δ∘ϕ$ as a composition of $ϕ$ and $δ$⁠, that is, $δ∘ϕ(x)=δ(ϕ(x))$ for any number $x$⁠. We say that $δ∘ϕ$ has a silent range for an interval $I$ if $δ∘ϕ(x)=0$ if $x∈I$⁠, and $δ∘ϕ(x)≠0$⁠, otherwise. For example, if we use the ReLU function as the activation function $ϕ$⁠, then $δ∘ϕ$ has a silent range for $I=(-∞,0]$ for any discretizer $δ$⁠. If we use the sigmoid function as the activation function $ϕ$ and linear partition as a discretizer $δ$⁠, then $δ∘ϕ$ has a silent range for $I=(-∞,tmax]$ where $tmax=ln(1/(2b-1))$ and $ln$ is the natural logarithm.

We define weight $w$ of $C$ as $w=22b$⁠, where $2b$ is the bit width possibly needed to represent a potential value invoked by a single input of a gate in $C$⁠.

Let $f:{0,1}[n]→{0,1}$ be a Boolean function. For a partition $(X,Y)$ of $[n]$⁠, we can view $f$ as $f:{0,1}X×{0,1}Y→{0,1}$⁠. We define communication matrix $MfX:Y$ as a $2|X|×2|Y|$ matrix where each row and column is indexed by $a∈{0,1}X$ and $b∈{0,1}Y$⁠, respectively, and each entry is defined as $MfX:Y(a,b)=f(a,b)$⁠. For $I⊆{0,1}X$ and $J⊆{0,1}Y$⁠, we call $R=I×J$ a combinatorial rectangle and say that $R$ is monochromatic with respect to $MfX:Y$ if $MfX:Y$ is constant on $R$⁠. If a circuit $C$ computes $f$⁠, we may write $MCX:Y$ instead of $MfX:Y$⁠. Figure 1a shows a communication matrix $MfX:Y$ where $f(ξ)= sign (∑i=16ξi-3)$⁠, $X={1,2,3}$ and $Y={4,5,6}$⁠. Figure 1b shows a monochromatic combinatorial rectangle $R=I×J$ with respect to $MfX:Y$⁠, where $I={(0,1,1),(1,0,1),(1,1,0)}$ and $J={(0,0,1),(0,1,0),(0,1,1),(1,0,1)}$⁠.

We also use well-known facts on the rank. Let $A$ and $B$ be two matrices of the same dimensions. We denote by $A+B$ the summation of $A$ and $B$⁠, and by $A∘B$ the Hadamard product of $A$ and $B$⁠.

In this section, we provide our main results showing the relationship among the four resources and trade-offs.

Let $C$ be a threshold circuit computing a Boolean function $f$ of $n$ variables. We prove the theorem by showing that for any partition $(X,Y)$ of $[n]$⁠, we can express $MCX:Y$ as a sum of matrices, each of which corresponds to an internal representation that arises in $C$⁠. Since $C$ has bounded energy, the number of internal representations is also bounded. We then show by the inclusion-exclusion principle that each matrix corresponding to an internal representation has a bounded rank. Thus, fact 1 implies the theorem.