Synaptic Information Storage Capacity Measured With Information Theory

In the late 19th century, Santiago Ramón y Cajal (1894) proposed that memories are stored at synapses and not through the generation of new neurons. Since then, there has been an extensive search for synaptic mechanisms responsible for learning and memory. Although synaptic plasticity is well established as an experience-dependent mechanism for modifying synaptic features, the precision of this mechanism is unknown. Connectomic studies are generally concerned with whether synapses exist between pairs of neurons, an important starting point; we are concerned here with quantifying synaptic strengths, the precision of synaptic plasticity, and the amount of information that synapses can store.

The existence of both intrinsic and extrinsic origins of variability and dysfunction of structural modulation (Kasai et al., 2021) motivates further exploration of the potential precision with which synaptic strengths can be adjusted. From an information theory perspective, there can be no information stored without precision. Indeed, the more precise that synaptic plasticity is, the more distinguishable synaptic strengths are possible and the greater the amount of information that can be stored at synapses. The synaptic strength is itself the information that is stored at a synapse, and this information is retrieved with subsequent synaptic transmission. Synapses are complex dynamical structures and “synaptic strength” encompasses intrinsic pre- and postsynaptic variables that can be modulated by synaptic plasticity. Among the variables are probability of presynaptic vesicular release, number of docked vesicles, number of postsynaptic receptors, and degree of short-term facilitation/depression, to name but a few. Thus, when a particular synapse is activated repeatedly, the resulting responses are expected to vary in amplitude in a way that is consistent with the synaptic strength (Kandaswamy et al., 2010; Klyachko et al., 2006).

Several studies have shown that synaptic strength is highly correlated with dendritic spine head volume (SHV; Harvey & Svoboda, 2007; Matsuzaki et al., 2004; Harris, 2020, reviewed in Yang & Liu, 2022). (Note that the abbreviations used in the article are listed at the end of section 1.) Pairs of dendritic spines on the same dendrite that receive input from the same axon (SDSA pairs) occur naturally in the brain and are expected to have experienced the same activation histories (Harris & Sorra, 1998). Hence, the precision of synaptic strength can be estimated by measuring the difference between the spine head volumes of individual SDSA pairs. These measurements can then be used to calculate the number of distinguishable synaptic strengths in a particular population of synapses.

Signal detection theory was used in previous studies to estimate the number of distinguishable synaptic strengths (Bartol et al., 2015, Bromer et al., 2018). Here, Shannon information theory was applied to overcome the limitations of using signal detection theory. Information theory is based on the distinguishability of messages sent from a transmitter to a receiver in the presence of noise. In the case of synaptic strength, the postsynaptic dendrite or soma is the receiver that must distinguish the strengths of messages coming from discrete synaptic inputs. The distinguishability of the received messages depends on the precision with which synaptic plasticity sets the strength of individual synapses. The new approach calculates the number of bits of Shannon information stored in synaptic strengths and thus quantifies empirically the synaptic information storage capacity (SISC).

The analysis of precision begins by measuring the coefficient of variation (CV) of SDSA pairs, as in Bartol et al. (2015). The new method differs in that nonoverlapping binning is performed across the entire range of sampled dendritic spines with the bin width, or precision level, equal to the median CV of the SDSA pairs. Comparison of the new SISC measurements with the previous results demonstrates that the new method is more robust to outliers and, importantly, can reveal gaps and variation in the shape of the distribution. Finally, in the new SISC analysis, the number of distinct synaptic strengths converges toward the actual (population) number of strengths as the number of spine head volumes increases and the true shape and extent of the distribution are sampled. Statistics and information theory allowed us to quantify the precision and information capacity of the message. Earlier versions of this new method appeared in Samavat et al., 2022a, 2022b; Samavat, Bartol, Harris et al., 2022; Samavat, 2023; Samavat et al., 2024).

The closeness of the observed value of SISC to the theoretical upper bound on SISC is defined as the benchmark distribution having maximum entropy, which occurs when there is a uniform distribution across the possible sizes. The KL divergence between the two discrete probability distributions does not depend on the choice of scale. Under these circumstances equation 2.3 reduces to log₂(N_S), where N_S is the number of bins. Thus, the maximum number of bits of Shannon information is calculated as H(U) = log₂(N_S), the entropy of a uniform distribution among N_S sizes. In this CA1 data set, N_S = 24 ± 5.40 (see Table 1) and sets an upper bound of 4.59 ± 0.37 bits for SISC (see Table 2, column 5), a value that is within the error range of the observed lower bound of 4.1 ± 0.39 bits.

The distribution of CA1 spine head volume bins was compared with a uniform distribution having 24 sizes (see Figure 2B and Table 2). The outcomes provide an upper bound on SISC for these sampled synapses but not necessarily for other synapses. We also calculated the ratio of the KL divergence values over the maximum value that KL divergence can possibly have (KL/KL_MAX) where KL_MAX equals the entropy of a uniform distribution, H(U). This ratio was then used to define the efficiency ratio η = (1 $-$ KL/KL_MAX) × 100. It measures the efficiency of the measured KL divergence value with respect to the maximum value it could hypothetically have (see Table 2, column 7). The higher the efficiency ratio, the more efficient is the use of distinguishable values for the storage of synaptic strength values across the population of synapses (see Figure 3). The KL divergence for this CA1 data set is 0.49 ± 0.068, a value suggesting that the number of distinguishable values approaches maximum entropy with an efficiency ratio of 89%. Thus, the reported distinguishable values are close to the maximum entropy.

This outcome provides a new statistic to assess information storage efficiency. This analysis for CV-width binned distribution of synaptic strength is more useful for comparisons between experimental conditions or between regions in diseased or normal brains. The precision level has a high impact on the efficiency ratio (on both KL in the numerator and KL_MAX in the denominator of η). Namely, cases with similar precision levels may still result in an insensitivity of the efficiency ratio to reflect deviations from a uniformly binned distribution. The efficiency ratio of binned distributions or the information storage capacity can be the same, while the shapes of the spine-head size distributions are different. Thus, when evaluating the information storage capacity of synaptic strength, both the entropy of synaptic strength and the efficiency ratio should be determined.

This article introduces a new analytical approach for determining SISC that has several advantages over our prior approach (Bartol et al., 2015). SDSA pairs are independent synapses, but because they are driven by the same input, they respond in a similar way and arrive at the same size due to the precision of similar underlying mechanisms in each synapse. Thus, the CV in the dimensions of these coactive pairs provides the basis for determining the range over which synaptic strength is essentially equal. There are more distinguishable strengths with greater Shannon information as the CV decreases due to higher precision. Bootstrapping analysis was used to estimate the standard error of the precision level, number of distinguishable strengths, and Shannon entropy and established SISC as a robust measure. The new method was tested on the same data from area CA1 of the hippocampus to compare it with the prior approach.

There are several advantages to the new SISC method for assessing the storage capacity of synapses. Signal detection theory (Bartol et al., 2015) assumed that the width of the gaussian curves, based on the CV of the SDSA pairs, was distributed equally along the full range of sampled spine head volumes, without accounting for gaps in the distribution. Thus, the signal detection theoretical approach tended to overestimate the true number of distinct synaptic strengths because the distribution of the population was assumed to be continuous. In the new SISC analysis, the number of distinct synaptic strengths defined by the individual bins converges toward the true number of sizes as the number of spine head volumes increases and the true shape of the distribution is sampled. A second advantage is that the full population of spine head volumes, not just the SDSA pairs, is included in the analysis, greatly improving the statistical power of the estimate. A third advantage is that there are no free parameters in the estimate, unlike signal detection theory where the degree of overlap of the gaussians is a free parameter. A fourth advantage is that the new method is robust to the outliers. The largest spine head volume we found in an earlier data set in rat hippocampal area CA1 was 0.55 cubic microns (Harris & Stevens, 1989). It is worth mentioning that by adding this one value to the 288 CA1 spine head volumes would result in 25 distinguishable strengths, an increment of 1 size using SISC. In contrast, using the prior signal detection approach results in 39 distinguishable gaussians spanning the range, a 50% increase in the number of sizes. Thus, the previous method is not robust to outliers. Finally, the new method, based on information theory, allows access to the frequency of the bins, making it possible to compute the entropy of the distinguishable synaptic strengths, the number of modes in the CV-width distribution, and any gaps in the range of functionally distinguishable synaptic strengths.

Information theory has been applied to analysis of spike trains (Dayan & Abbott, 2005) but has not been used at the level of synaptic strength. We have shown that the amount of information represented by synaptic strengths in neural circuits can be quantified by the distinguishability of synaptic dimensions. Here distinguishability fundamentally depends on the precision of the synaptic strength measurements and histories of activation. When the precision of synaptic strengths is low, the amount of information that can be stored in the ensemble of the neurons will also be low. Complete absence of precision implies a random process for setting synaptic strength and no information being stored at synapses. Because spine head volume is highly correlated with synapse size (Bartol et al., 2015), the precision of spine head volumes can be used to assess the distinguishability of the synaptic strengths. High precision yields a greater number of distinguishable spine head volume bins and hence higher information storage capacity. The number of distinguishable strengths is not static but varies with the history of synaptic plasticity and is different in different parts of the brain. Thus, the amount of information that a population of synapses can store is not fixed but can be changed.

Comparison with the uniform distribution was made because it is the most conservative assumption when anatomical constraints on the spine head volume distribution are not known. When no constraint is applied (except the number of sizes being fixed), the uniform distribution has maximal entropy among every discrete distribution. This is why the uniform distribution for a fixed number of sizes is a lower bound on optimality and an upper bound for SISC. For example, the entropy of the observed synaptic strength distribution in area CA1 based on the probability of the 24 distinguishable sizes was 4.1 bits, while the maximal entropy for the same number of sizes is the uniform distribution with 4.6 bits of information. Thus, the KL divergence analysis provided a reliable measure of the closeness of the CV-width binned distributions of spine head volume to the maximum entropy distribution with an efficiency ratio of 89%. The amount of information stored across 24 sizes is equivalent to the maximum information that could hypothetically be stored across 17 sizes. The closer these two numbers are, the more efficient is the use of the distinguishable synaptic strengths.

If biological constraints were found at some point in the future, the maximal entropy distribution could be calculated under those constraints, and the distance between the empirical distribution (such as log normal) and maximal entropy distribution (with optimized parameters) could be determined. For example, if one calculates the maximal entropy distribution as a log normal distribution with well-constrained shape parameters (i.e., biological constraints on $μ$ and σ), under those conditions the KL divergence would be different.

It is worth noting that synaptic activity, and hence SISC, is highly variable and changes during an animal's behavior. This article is concerned only with the optimality of information storage capacity based on proxies for synaptic strength as the unit of storage of information and the distribution of strengths at a single moment of time. This analysis sets a lower bound on the information storage capacity of synapses. Other sources of storage capacity that include biochemical and structural substrates across timescales could augment our estimate of long-term plasticity based on spine head sizes. Another question is how working memories are stored in neural circuits and how efficiently the synaptic strength is used in those codes. These issues are beyond the scope of this study and are open for future research.

Information storage and coding in neural circuits have multiple substrates over a wide range of spatial and temporal scales. How information is coded and its potential efficiency depend on how the information is stored. The efficiency of information storage was analyzed by comparing the distribution of synaptic strengths, as measured in spine head volumes, with the uniform strength distribution, the maximum entropy distribution. The outcomes reveal a near maximal efficiency in storing information across the distinguishable sizes of spine head volumes. These new measures of precision and efficiency can be used to assess the impact on SISC of other factors, such as pre- or postsynaptic dimensions and composition and variation across brain regions and species. It will be possible to determine effects on SISC of intrinsic differences or changes in the location or dimensions of key subcellular components, such as mitochondria, smooth endoplasmic reticulum, polyribosomes, endosomes, and perisynaptic astroglia. These opportunities to generalize SISC will result in greater understanding of the role of each component individually, and in concert, in determining the precision and efficiency of synaptic strength and plasticity. Ultimately, the outcomes should give new insight into how disrupted synapses result in cognitive decline during aging or as a consequence of neurological disorders.

The CA1 data set was published in Harris et al. (2015) and Kinney et al. (2013), and reanalyzed in Bartol et al. (2015) and this article.

The 3DEM data set that was used in this study was reconstructed from serial thin sections in the middle of the stratum radiatum of hippocampal area CA1 from an adult male rat (55–65 days old) (Mishchenko et al., 2010; Bourne et al., 2013). The dense reconstruction of 6 $×$ 6 $×$ 5 $μm3$ of hippocampal neuropil reanalyzed in this study was originally processed as previously described in a study of the extracellular space (Kinney et al., 2013). In this data set, authors in Bartol et al. (2015) identified 10 axon-coupled synaptic pairs. To perform an accurate and robust geometric analysis of the dendrites, dendritic spines, and axons, the reconstructed surface meshes for artifacts were corrected by Bartol et al. (2015) to be computational-quality meshes as described elsewhere (Kinney et al., 2013; Edwards et al., 2014). The postsynaptic densities (PSDs) and presynaptic active zones (AZs) were identified in the serial section transmission electron microscopy (ssTEM) images by their electron density and the presence of closely apposed presynaptic vesicles. A total of 449 synaptic contacts were found in the dense reconstructed volume of neuropil. Bartol et al. (2015) excluded a number of synapses from the analysis if they were partially clipped by the edge of the data set (142) or were shaft synapses (20), leaving 287 valid synapses on dendritic spines in the dense model.

Bartol et al. (2015) used Blender's functionality, which is user extensible via a Python interface for creating add-ons. Bartol et al. created a Python add-on for Blender that enabled the selection of the mesh triangles of the dendrite corresponding to the spine head and whole spine of each individual spine. The add-on tagged each selected set of triangles with metadata for the spine name and geometric attributes of the head, whole spine, and neck as described below. The selection of the spine head was made by hand based on a standardized procedure in which the junction between the head and neck was visually identified as halfway along the concave arc as the head narrows to form the neck (see Figure 3 and the figure supplement 1A in Bartol et al., 2015). To select the whole spine, a similar visual judgment was made to locate the junction where the neck widens as it joins the dendritic shaft. Once the appropriate area was selected, the tool was designed to automatically create the convex hull of the selected region. The closed mesh formed by the Boolean intersection of the convex hull and the dendrite was used to determine the measured volume of the spine head or whole spine. The volume of the neck was calculated by taking the difference between these two measurements. Areas were computed from the selected regions for spine head and whole spine.

The codes for algorithms used in this study will be available in the following github link upon publication: https://github.com/MohammadSamavat.

The standard error of median for the precision levels of each of the CA1 data sets’ SDSA pairs is calculated with algorithm 1. The idea is to generate 1000 bootstrap samples of length n, each sampled from the n SDSA pairs with replacement, to estimate the standard error of median for the n SDSA pairs (see Table 1, column 3). The standard error of the median of spine head volumes follows the same procedure using algorithm 1.

To construct the bins, spine head volumes are sorted from smallest to the largest. The first value (smallest value) is selected and the CV of that value and the remaining head volumes are calculated in a pairwise manner. The head volumes for which the calculated CV is below the threshold (the median value of the SDSA pairs CV) are assigned to the first bin and deleted from the pool of N spine head volumes. This procedure is repeated until the CV exceeds the median SDSA pairs CV and a new bin is formed. New categories are formed until all the remaining spine head volumes are assigned to a bin and the original vector of spine head volumes is empty (see algorithm 2 for details). It is guaranteed that the coefficient of variation between each random pair within each bin is less than the threshold value measured from the reconstructed tissue SDSA pairs. All spine head volumes are rounded to two significant digits for the display.

For Figure 2B, the y-axis shows the percentage of spine head volume counts in the respective bin. The x-axis shows the spine head volumes in $μm3$ on the log scale. The width of each bin is exactly the median value of the set of CV values plotted in Figure 1D (example: for bin-1 [x1,x2], CV(x1,x2) = 0.12, where x1 is the smallest spine head volume in CA1 and x2 is a larger hypothetical head volume that has a CV of 0.12 with x1). The height of bin-1 is the number of spine head volumes in that range normalized to the total number of spine head volumes in this data set (288).

Shannon's information theory is the rigorous basis for a new method to quantify empirically SISC—that is, the number of bits of Shannon entropy stored per synapse. For the new method, first the precision of SDSA pairs was measured from the coefficient of variation (CV) of the pairs (illustrated in Figure 1D, also used in Bartol et al., 2015). The measured precision was then used as a binning threshold for nonoverlapping bin analysis (see Figure 2B and appendix Figure 2) using algorithm 2. This analysis yielded the number of distinguishable sizes of spine head volumes in the sample of reconstructed dendrites.

Next, the Shannon information per synapse is calculated by using the frequency of spine head volumes in the distinguishable values where each bin is a different message for the calculation of Shannon information. The maximum number of bits is calculated as the log₂(N_S) where N_S is the number of sizes or categories that set an upper bound for the SISC.

When calculating the amount of entropy per synapse, the random variable is the synapse size, and the number of distinguishable values is the realization of a random variable for the occurrence of each size. The probability of the occurrences of each size is calculated by the fraction of the number of spine head volumes in each of the bins over the total number of spine head volumes in the reconstructed volume.

The information coding efficiency at synapses is measured by KL divergence to quantify the difference between two probability distributions—one from the categories of spine head volumes and the other from a corresponding uniform distribution with the same number of categories.

The statistical analysis and plots were generated using Python 3.4 with NumPy (Harris et al., 2020), SciPy (Virtanen et al., 2020), and Matplotlib (Hunter, 2007) and R package ggplot2 (Wickham, H., 2011).

For the precision analysis, we used the coefficient of variation (see equation 2.1) as a metric to show the precision level by calculating the ratio of standard deviation (see equation 2.2) over the mean of N joint synapses. Here N is 2 but can take higher values as up to 7 have been detected in previous studies. Since this is a sample from the unknown population of joint synapses, we used the corrected standard deviation formula with 1/(N$-$ 1) factor.

Bootstrapping was done by combining algorithms 1 and 2 to calculate the standard errors as explained in sections 5.4 and 5.5.

The standard errors of the entropy, efficiency ratio, maximum entropy for uniform distribution, and KL divergence (see Table 2, columns 2–5) are all calculated similarly using the bootstrapping technique explained in algorithm 1. (See Efron et al., 2021, for further information regarding bootstrapping for the calculation of standard error.)

M.S., T.M.B., K.M.H., and T.J.S. designed research, analyzed data, and performed research. M.S. designed and implemented all simulation algorithms and generated results for the manuscript and applied the information theory to the analyses with contributions from T.M.B., K.M.H., and T.J.S., M.S. and T.J.S. wrote the article with contributions from T.M.B. and K.M.H.

We thank Adel Aghajan and Wenxin Zhou for helpful discussion regarding information theory and bootstrapping analysis and Patrick Parker for editorial support. Also, we thank Cailey Bromer for her help in determining some of the dendritic spine head volumes. Feedback from the reviewer greatly helped us clarify the terminology and the conceptual framework in this article. This research was supported by NSF DBI-1707356; DBI-2014862; NSF 2219894 NSF IIS-2219979; NIH P41GM103712; NIH MH095980-07; NIH MH115556; NIH MH129066.