Divisive normalization has been proposed as a nonlinear redundancy reduction mechanism capturing contrast correlations. Its basic function is a radial rescaling of the population response. Because of the saturation of divisive normalization, however, it is impossible to achieve a fully independent representation. In this letter, we derive an analytical upper bound on the inevitable residual redundancy of any saturating radial rescaling mechanism.
This simple but elegant mechanism is so apt in capturing the behavior of neurons throughout the brain that it has rececently been termed canonical computation (Carandini & Heeger, 2011). One possible computational goal of this widespread mechanism could be the reduction of redundancies among neural responses to natural signals in accordance with Barlow's idea of redundancy exploitation (Barlow, 1961, 2002) as demonstrated in the seminal paper by Schwartz and Simoncelli (2001).
2. Analytical Results
After whitening, natural image patches can be well modeled by Lp-spherically symmetric distributions (Sinz & Bethge, 2009). These can be transformed into factorial distribution by a nonlinear radial rescaling similar to divisive normalization (see Figure 1a). Since the radial rescaling of divisive normalization saturates at , it cannot achieve full redundancy reduction. Numerical computations show that the multi-information rate of a radially truncated p-generalized normal distribution is monotonicly decreasing with the truncation threshold (see Figure 1b). Therefore, the limiting case provides an upper bound on the information rate for arbitrary radially truncated p-generalized normal distributions. This upper bound is given by the multi-information rate of the uniform distribution within the Lp-unit ball that we derived here. It turns out that the upper bound is quite low compared to a lower bound on the multi-information rate of natural images of nats/pixel (Hosseini, Sinz, & Bethge, 2010; see also Figure 1c). This means that the dependencies due to radial truncation are negligible compared to the dependencies present in unnormalized natural images. Therefore, the multi-information of the uniform distribution on the Lp-unit ball can serve as a meaningful lower bound on the redundancy reduction that radial rescaling mechanisms should be able to achieve at least.
This work was supported by the Bernstein Center for Computational Neuroscience (FKZ 01GQ1002) and the German Excellency Initiative through the Centre for Integrative Neuroscience Tübingen (EXC307). Fabian Sinz wants to thank Oleksandr Pavlyk for helpful discussions on generalized hypergeometric functions.