Exploring Trade-Offs in Spiking Neural Networks

Over the past decade, deep neural networks (DNNs) have become indispensable tools in statistical machine learning, achieving state-of-the-art performance in various applications, including computer vision (Krizhevsky et al., 2012; Szegedy et al., 2013), natural language processing (Vaswani et al., 2017; Devlin et al., 2018; Brown et al., 2020), and reinforcement learning (Mnih et al., 2013, 2016). However, their impressive performance often comes at a significant hardware and energy cost. For example, natural language processing models can consist of billions of parameters and require energy-intensive GPU clusters to train efficiently (Brown et al., 2020). These hardware and energy requirements pose a significant challenge in terms of sustainability and restrict the practical applicability of DNNs in resource-limited environments such as low-powered edge devices. Therefore, exploring more energy-efficient alternatives to DNNs is crucial not only to address the environmental cost of machine learning but also to provide practical and sustainable solutions in edge computating.

One possible alternative to DNNs is spiking neural networks (SNNs). Spiking neurons process information through discrete spatiotemporal events known as spikes rather than continuous real-number values (Maass, 1997; Gerstner & Kistler, 2002). Spikes enable efficient implementations of neural networks on non-von Neumann neuromorphic hardware such as Intel Loihi, IBM TrueNorth, Brainscale2, and SpiNNaker (Painkras et al., 2013; Akopyan et al., 2015; Schmitt et al., 2017; Davies et al., 2018; Furber, 2016; Hendy & Merkel, 2022). Such hardware consumes only a fraction of the power required by DNNs on von Neumann computers and thus represents a suitable solution for energy-efficient edge computing (Blouw et al., 2019; Taunyazov et al., 2020).

The power consumption of neuromorphic hardware is closely related to the number of spikes they produce. Therefore, sparse SNNs that fire only a small number of spikes achieve high energy efficiency on hardware. However, such networks also transmit less information, creating a trade-off between energy consumption and model accuracy. In addition to sparsity, various trade-offs between performance and other aspects of SNNs are commonly explored in the literature (Park et al., 2021; Yin et al., 2023; Li et al., 2021; Diehl et al., 2015).

For instance, unsupervised learning rules such as spike time dependent plasticity (STDP) can be implemented directly on neuromorphic hardware, allowing for biologically plausible and energy-efficient training of SNNs that are resilient to substrate noise of analog circuits (Kim et al., 2020). However, the performance of unsupervised learning lags behind that achieved with supervised learning. Meanwhile, state-of-the-art performance with SNNs is currently achieved through various error backpropagation (BP) techniques adapted from deep learning (Bohté et al., 2000; Lee et al., 2016; Jin et al., 2018; Kheradpisheh & Masquelier, 2020; Kheradpisheh et al., 2021; Zhang et al., 2022; Shrestha & Orchard, 2018; Wu et al., 2018; Mostafa, 2016; Comsa et al., 2020; Göltz et al., 2021; Fang et al., 2021). However, BP algorithms often require constraints on spikes to achieve high sparsity and low prediction latency, at the cost of performance (Yan et al., 2022; Mostafa, 2016; Guo et al., 2020; Kheradpisheh & Masquelier, 2020; Kheradpisheh et al., 2021; Göltz et al., 2021). In addition, BP requires global transport of information that is incompatible with neuromorphic hardware, and training must be performed either offline or in-the-loop, where a conventional computer is used in conjunction with neuromorphic hardware (Schmitt et al., 2017; Göltz et al., 2021). Therefore, SNNs trained offline or in the loop must also be resilient to the substrate noise and weight quantization of analog hardware to avoid performance drops at deployment.

One particularly interesting approach for training fast, energy-efficient, and noise-resilient SNNs is Fast & Deep (Göltz et al., 2021). This exact BP method employs time-to-first-spike (TTFS) coding, which restricts neurons to fire only once. Inspired by the human visual system (Thorpe et al., 1996), TTFS is based on the idea that first spikes of neurons must carry most of the information about input stimuli, enabling fast, sparse, and energy-efficient computation. Due to this constraint imposed on firing, we thus referred to TTFS networks as constrained SNNs. However, relaxing the spike constraint of TTFS and allowing multiple spikes per neuron typically results in higher information rates, better performance, and increased noise resilience compared to TTFS networks (Jin et al., 2018; Zhang & Li, 2019; Shrestha & Orchard, 2018; Lee et al., 2016; Zhang et al., 2022). One might assume that such network, referred to as unconstrained SNNs, would improve performance and noise robustness but also result in slower inference and lower energy efficiency due to increased firing rates.

In this work, we explore the trade-offs among performance, convergence, energy consumption, prediction speed, and robustness of SNNs, with and without the spike constraint imposed by TTFS. We make the following main contributions:

• We demonstrate that many aspects are driven by the weight distribution in Fast & Deep, highlighting trade-offs among performance, energy consumption, latency, and stability in TTFS SNNs.

• We extend the Fast & Deep algorithm to multiple spikes per neuron and describe how errors are backpropagated in unconstrained SNNs.

• We show that our proposed method improves performance while providing better convergence rate, similar sparsity, comparable latency, and improved robustness to noise compared to Fast & Deep, suggesting that relaxing the spike constraints in TTFS can lead to better trade-offs.

In this section, we describe our generalization of the Fast & Deep algorithm to multiple spikes per neuron. Our main contribution lies in the reset of the membrane potential and how errors are backpropagated through interneuron and intraneuron dependencies.

We consider a neural network of current-based leaky integrate-and-fire neurons with a soft reset of the membrane potential. (Gerstner & Kistler, 2002; Davies et al., 2018; Göltz et al., 2021).

The reset of the membrane potential in equation 2.1 is the major difference with the TTFS model used in Fast & Deep (Göltz et al., 2021). By resetting the membrane potential after postsynaptic spikes, our model allows for further integration of inputs and thus the firing of several spikes. Therefore, this relaxes the constraint on spike counts imposed by Fast & Deep.

This equation can be thus used to infer the spike trains of neurons in an event-based manner.

Because the spike timing now has a closed-form solution, it becomes differentiable, which allows the computation of an exact gradient. We first state the total change of weight between two neurons.

In this section, we compare our proposed method with Fast & Deep. This evaluation was conducted based on multiple criteria, including performance on benchmark data sets, convergence rate, sparsity, classification latency, as well as robustness to noise and weight quantization.

Experimental conditions were standardized for both methods except for weight distributions and thresholds. Two uniform weight distributions, (w_{i, j} ∼ U(− 1, 1) and w_{i, j} ∼ U(0, 1)), were used to evaluate Fast & Deep, to measure the effect of initial weight distributions on the different evaluation criteria. Our method was solely assessed using w_{i, j} ∼ U(− 1, 1), as positive initial weights lead to excessive spiking activity, hindering computational and energy efficiency. Thresholds were manually tuned to find the best-performing networks and kept fixed during training. In our experiments, all layers (including convolutional layers) have been directly trained using our proposed method and Fast & Deep, and no conversion from DNN to SNN has been performed. More details about our experimental settings are in the appendixes.

To assess the performance of our proposed method, we trained fully connected SNNs on the MNIST (LeCun et al., 2010), EMNIST (Cohen et al., 2017) (Balanced Extended MNIST), and Fashion-MNIST (Xiao et al., 2017) data sets as well as convolutional SNNs on MNIST. We also evaluated our method on temporal data classification by training fully connected networks on the Spiking Heidelberg Digits (SHD) data set (Cramer et al., 2022), an English and German spoken digits classification task. We compared our results to those obtained using the original Fast & Deep algorithm. Table 1 summarizes the average test accuracies of both methods given the considered initial weight distributions. For completeness, a comparison of Fast & Deep and our method with other spike-based BP algorithms is in the appendix.

First, it should be noted that the Fast & Deep algorithm generally achieves better performance when the weights are initialized with positive values, which is consistent with the choice of weight distribution made by Göltz et al. (2021). Second, our proposed method demonstrates superior performance compared to the Fast & Deep algorithm, with improvement margins ranging from 1.05% on MNIST to 2.41% on the more difficult EMNIST data set. This is not surprising given that unconstrained SNNs are known to perform better compared to TTFS networks (Jin et al., 2018; Zhang & Li, 2019; Shrestha & Orchard, 2018; Lee et al., 2016; Zhang et al., 2022). Moreover, our method outperforms by at least 19% Fast & Deep on the SHD data set with a single hidden layer of 128 neurons. Note that no recurrent connections were used in these experiments. To understand the reason for this performance gap, we analyzed the spiking activity in the hidden layers after training. Figure 3 shows that TTFS neurons respond only to early stimuli. In temporal coding, high-valued information is encoded by early spikes. Training with a one-spike constraint is therefore energy efficient but tends to make SNNs spike as early as possible, thus missing the information occurring later in time. In contrast, neurons trained without spike constraint are able to respond throughout the duration of the sample, thus capturing all the information despite an increased number of spikes fired. This highlights the importance of firing more than once and demonstrates that a trade-off exists between performance and energy consumption when processing temporal data.

In addition, our results indicate that the convergence rate of SNNs with multiple spikes is higher compared to TTFS networks. Figure 2 depicts³ the evolution of the test accuracy of both methods on MNIST. This demonstrates that our method can reach desired accuracies in fewer epochs compared to Fast & Deep. Such improvement implies that discriminative features are learned earlier, during training.

Achieving high sparsity in trained SNNs is critical for energy efficiency, as neuromorphic hardware consumes energy only at spike events.

Figure 4 shows the population spike counts of fully connected SNNs trained on MNIST using both methods. It appears that the initial weight distribution plays an important role in the final sparsity level of the trained SNNs. In the analyzed case, SNNs initialized with positive weights appear to be less sparse after training than SNNs initialized with both negative and positive weights.

While our proposed method allows for an increased number of spikes per neuron, which implies more energy consumption, we found that it can achieve similar sparsity as TTFS networks initialized with both negative and positive weights while performing better than TTFS networks initialized with positive weights only (see Figure 4 and Table 1). This suggests that our proposed method can offer improved trade-offs between accuracy and sparsity.

To understand why our method can achieve such levels of sparsity despite not imposing any constraints on neuron firing, we analyzed the average activity in each network. Figure 5 shows that neurons trained with Fast & Deep fire indiscriminately in response to any input digit, which is characteristic of temporal coding where information is represented by the timing of spikes rather than the presence or absence of spikes. In contrast, SNNs trained with our method exhibit a different distribution of firing activity, with certain key neurons selectively responding to specific digits. Figure 4 indicates that only 7% of neurons trained with our method are active during inference (i.e., neurons that fire at least once). In comparison, the SNN trained with Fast & Deep and initialized with both negative and positive weights has 24% of its neurons firing. Therefore, the reduced proportion of active units in our method compensates for the increased number of spikes per neuron, leading to fewer spikes emitted in the network.

We found during our experiments that the sparsity of SNNs trained with our method was also influenced by the choice of threshold values. More precisely, we observed that decreasing the output threshold resulted in a reduction of activity in the network, as illustrated in Figure 6a. This decrease in activity occurred because the output neurons fired more often, which is illustrated in Figure 6b. The increased number of output spikes caused the loss function to produce more negative errors, which resulted in negative changes of weights in the hidden layer. However, lowering the threshold has a dual impact on activity: it increases the firing rates at initialization but also contributes to producing more negative errors, which can decrease activity during learning. While controlling sparsity using thresholds is trivial in shallow SNNs, a fine balance between thresholds has to be found to control sparsity in Deep SNNs, making it challenging to achieve sparsity with larger architectures. However, our findings suggest that threshold values play a crucial role in determining the sparsity of unconstrained SNNs and can be seen as a way to control activity without the need for firing rate regularization.

We also investigated the prediction latency of SNNs trained using different methods. This corresponds to the amount of simulation time the models need to reach full confidence in their predictions and determines the simulation duration required to achieve high accuracy.

Figure 7a shows the averaged prediction confidence over time for SNNs trained using each method. We found that when initialized with only positive weights, Fast & Deep achieved high confidence on predictions after more than 150 ms. However, when initialized with both negative and positive weights, both Fast & Deep and our proposed method achieved confidence earlier (in 20 ms and 50 ms, respectively). This suggests that initialization has a significant impact on prediction latency. However, SNNs trained with our proposed method are slightly slower than TTFS networks trained with Fast & Deep for the same weight distribution due to the increased spike count per neuron.

Despite this difference, the short latency of these networks allows for shorter simulation durations, which can further improve sparsity without affecting performance. In Figure 7b, we show the relationship between population spike count and accuracy for each SNN as simulation time increases from 0 to 200 milliseconds. This demonstrates that by reducing the simulation time, SNNs can become sparser while maintaining high performance. Therefore, the sparsity-accuracy trade-off can be further improved after training by adjusting the simulation duration. This also demonstrates that our method can align with the level of the sparsity of Fast & Deep while still performing better.

Analog neuromorphic hardware is inherently noisy and often limited to specific ranges and resolutions of weights. Having a model that is robust to noise and weight quantization is therefore important to achieve high performance on such systems. To assess the robustness of each method, we measured the impact of spike jitter, weight clipping, and weight precision on their accuracy. In these experiments, performance was normalized with the maximum accuracy of each method to better compare variations.

Figure 8a shows the impact of spike jitter on the performance of each method. These results were produced by artificially adding noise to spike timings with a normal distribution $N(0,σ)$ during training. When initialized with both negative and positive weights, Fast & Deep appears to be less robust than using only positive weights. With negative weights, only a fraction of neurons transmits information, which leads to an increased sparsity, as illustrated in Figure 4. Therefore, introducing noise to spike timings has a significant impact on performance. In contrast, positive weights ensure consistent network activity and redundancy in transmitted information. SNNs initialized with positive weights are thus less affected by spike jitter. However, SNNs trained with Fast & Deep remain susceptible to noise, even with positive initial weights. Perturbations in spike timing still have a critical impact on temporal coding. In contrast, our proposed method demonstrates greater robustness to spike jitter than Fast & Deep, with minimal variation observed. This is a result of the redundancy created by the multiple spikes fired by neurons.

In Figure 8b, we demonstrate the effect of weight range on performance by clipping weights between the range [− w_clip, w_clip] during training. The performance of Fast & Deep initialized with positive weights degrades when w_clip is lower than 1.5. However, both our method and Fast & Deep exhibit robustness to reduced weight ranges when initialized with both negative and positive weights. This suggests that weight distribution may play a role in the network’s resilience to limited weight ranges.

Finally, Figure 8c shows the performance of each method with reduced weight resolutions from 5 to 2 bits (results with float precision are also given as a reference). It highlights that Fast & Deep is less robust to reduced weight precision than our method, particularly with negative weights. In contrast, our approach is only slightly affected by the decreased precision, even when reduced to as low as 2 bits.

In this work, we explored the trade-offs between performance and various aspects of TTFS SNNs such as sparsity, classification latency, and robustness to noise and weight quantization. We also generalized the Fast & Deep algorithm by incorporating a reset of the membrane potential, which enables multiple spikes per neuron, and compared the improvements of the proposed method with the origin algorithm on those trade-offs.

We found that initializing Fast & Deep with positive weights leads to better generalization capabilities compared to initializing with both negative and positive weights. This observation is consistent across the benchmarked data sets, as shown in Table 1. However, relaxing the spike constraint improves the overall performance and convergence rate of SNNs, at least on the benchmark problems we considered. This result was expected before our experiments since BP methods that use multiple spikes per neuron generally perform better than methods that impose firing constraints (Jin et al., 2018; Zhang & Li, 2019; Shrestha & Orchard, 2018; Lee et al., 2016; Zhang et al., 2022).

Our experiments also demonstrate that the weight distribution significantly influences the sparsity of Fast & Deep. We observed that SNNs with positive weight initialization tend to be less sparse than those initialized with weights between −1 and 1. However, the former consistently outperforms the latter in terms of performance. This highlights the accuracy-sparsity trade-off often observed when training SNNs (Yin et al., 2023; Li et al., 2021). The quasi-dense activity provided by positive weights explains the difference in sparsity, as shown in Figure 5b. In contrast, initializing Fast & Deep with both negative and positive weights leads to fewer active neurons due to the inhibition provided by negative weights. Additionally, neurons trained with Fast & Deep fire indiscriminately to stimuli, suggesting a pure temporal representation of information, whereas neurons trained with our proposed method selectively respond to their inputs and exhibit a different distribution of activity, as shown in Figure 5c. Our unconstrained SNNs allow for a different distribution of the spike activity, whereby key neurons can fire more often than others, while irrelevant neurons may not spike at all. This enables our method to achieve a level of sparsity comparable to Fast & Deep on image classification, as illustrated in Figure 4.

To achieve a high degree of sparsity without firing rate regularization, thresholds can be tuned to indirectly influence spiking activity through learning. Decreasing thresholds increases the firing rate of downstream layers, resulting in more negative errors at outputs and consequently negative weight changes in hidden layers, as shown in Figure 6. This mechanism offers a natural way to control sparsity in unconstrained SNNs without requiring any regularization technique. By leveraging this principle, we were able to train shallow SNNs with a sparsity level similar to Fast & Deep while achieving higher performance. This implies that allowing multiple spikes per neuron has the potential to enhance the accuracy-sparsity trade-off and prompts further investigation into the effectiveness of TTFS in achieving efficient computation. However, finding thresholds that lead to high sparsity is more difficult when networks become deeper due to the fluctuations in the firing rates of each layer. The factors that influence sparsity in unregularized SNNs are currently not fully understood and present an opportunity for future research to investigate how to naturally achieve sparsity in deep architectures.

Our experiments on the SHD data set also demonstrated the significance of the accuracy-sparsity trade-off when processing temporal data. We found that the ability of neurons to fire multiple spikes is critical in capturing all the information about the inputs over time. However, since TTFS neurons encode important information in early spikes, they tend to fire too early to capture all the information, which makes them less effective than unconstrained SNNs on temporal data. Although TTFS SNNs are more energy efficient, the relaxation of spike constraints in unconstrained SNNs allows neurons to fire throughout the simulation, thereby capturing all the relevant information. Consequently, they perform significantly better than TTFS SNNs when processing temporal data.

In addition to performance and sparsity, we measured the prediction latency of each method, which is the waiting time required before the system can make reliable predictions. We found that the speed of classification was primarily driven by the weight distribution. Figure 7a shows that both Fast & Deep and our method achieve similar latencies when initialized between −1 and 1, with a slight advantage for Fast & Deep. However, when initialized with only positive values, Fast & Deep requires more simulation time to achieve full confidence in predictions. Low latency is advantageous not only in terms of inference speed but also in improving energy efficiency. If predictions occur early enough, the duration of simulations can be significantly reduced, limiting the number of spikes fired by neurons. In Figure 7b, we demonstrated that reducing the simulation time can lead to a reduction in computational cost for both Fast & Deep and our method while maintaining the same performance. This shows that prediction latency, energy consumption, and performance are closely related and that unconstrained SNNs can offer better trade-offs between these aspects than TTFS SNNs.

The last characteristic that we investigated is the robustness to the noise and weight quantization inherent in analog neuromorphic hardware. The timing of spikes is a critical factor for the performance of TTFS SNNs as it carries most of the information. Therefore, perturbations in these timings and weight constraints can significantly affect the reliability of the feature extraction. In contrast, our proposed method benefits from an increased number of spikes per neuron, providing redundancy that enhances resilience to noise and weight constraints. For instance, Figure 8a demonstrates that our method is less affected by perturbations in spike timings than Fast & Deep. This suggests that our proposed method has the potential to provide more stable learning on analog neuromorphic hardware than Fast & Deep.

Finally, our work specifically concentrated on backpropagation in SNNs with fixed thresholds and time constants. However, several studies have demonstrated that incorporating adaptive thresholds and trainable time constants can enhance the convergence, sparsity, and performance of SNNs (Zambrano et al., 2019; Chen et al., 2022; Fang et al., 2021; Yin et al., 2021). Therefore, future studies could explore the integration of adaptive thresholds and trainable time constants into our proposed method to further enhance the sparsity-accuracy trade-offs in SNNs.

Our work demonstrates that relaxing the spike constraint of TTFS SNNs results in improved trade-offs among performance, sparsity, latency, and noise robustness. Our findings also highlight the crucial role of thresholds in regulating the sparsity of unconstrained SNNs during learning, which could serve as a natural alternative to firing rate regularization. Although error backpropagation algorithms for SNNs are incompatible with neuromorphic hardware, their development provides valuable insights into how spiking neurons affect objective functions and could support the development of hardware-compatible algorithms. Therefore, our work contributes to a better understanding of how to compute exact gradients in SNNs and highlights the advantages of using multiple spikes per neuron over TTFS.

We implemented the method introduced in section 2 in a custom Python simulator for GPUs using CuPy (Okuta et al., 2017). In our implementation, both simulations and error backpropagation are event based.

We used a TTFS encoding scheme to benefit from a low number of input spikes and fast processing. For image classification tasks, we encoded the pixels into spike timing as follows:

To reproduce Fast & Deep with TTFS models, we constrained the number of firings allowed per neuron to one in our implementation and used a TTFS softmax cross-entropy loss function, as described in (Göltz et al., 2021).

For the MNIST EMNIST and SHD data sets, we trained both TTFS and unconstrained fully connected SNNs with a batch size of 50 and a maximum number of spikes per neuron of 30 for the unconstrained SNNs. Output spike targets were set to 15 for the target label and 3 for the others. We used a learning rate of λ = 0.003 for image classification and λ = 0.001 for the SHD data set. No data augmentation was used with full connected networks.

For Fashion-MNIST, we implemented a three-layer, fully connected network composed of two hidden layers of 400 neurons each and a 10 neuron output layer. We allowed a maximum number of spikes per neuron of 5 for the hidden layers and 20 for the output layer. We also set the target spike counts to 15 for the true class and 3 for the others. We used a batch size of 5 with a learning rate of λ = 0.0005, a learning rate decay factor of 0.5 every 10 epochs, and a minimum rate of 0.0001.

The weight kernels of convolution neurons were shared within each layer, as in rate-based CNNs. Convolution allows the detection of spatially correlated features and therefore makes networks invariant to translations. In contrast to fully connected SNNs, the translation invariances of CSNNs allow the networks to detect objects at different locations in space. We used a six-layer CSNN composed of two spiking convolutional layers of 15 5 × 5 and 40 5 × 5 filters, respectively, each followed by 2 × 2 spike aggregation pooling layer (i.e., the spike trains of input neurons are aggregated into a single spike train). The spikes of the last pooling layer are finally sent to two successive fully connected layers of sizes 300 and 10, respectively. Each layer allows an increasing number of spikes per neuron, starting from a single spike for the first convolutional layer, 3 for the second layer, 10 for the fully connected layer, and 30 spikes per neuron for the output layer. We also set the output spike targets to 30 for the true label and 3 for the others. The CSNN was also trained with data augmentation. In this case, we used elastic distortions (Simard et al., 2003) to transform the MNIST training images. We finally trained the networks for 100 epochs with a batch size of 20, a learning rate of λ = 0.003, a decay factor of 0.5 every 10 epochs, and a minimum rate of 0.0001.

In all our experiments, we used the Adam optimizer with the values of β₁, β₂, and ϵ set as in the original paper (Kingma & Ba, 2015). Initial weights were randomly drawn from a uniform distribution U[a, b]. Networks trained on image classification used the same base time constant of τ_s = 0.130. For the SHD data set, we used a time constant of τ_s = 0.100. All thresholds were manually tuned to find the best-performing networks for each method and data set. Thresholds were then kept fixed during training. Finally, we did not use any regularization or synaptic scaling techniques in any of our experiments to provide a fair comparison between TTFS and unconstrained SNNs.

The code produced in this work will be made available at: https://github.com/Florian-BACHO/bats