Online fitting of computational cost to environmental complexity: Predictive coding with the ε-network

We propose the Epsilon Network (ε-network), a network that automatically adjusts its size to the complexity of a stream of data while performing online learning. The network optimises its topology during training, simultaneously adding and removing neurons and weights: it adds neurons where they can raise performance, and removes redundant neurons while preserving performance. The network is a neural realisation of the ε-machine devised by Crutchfield and al. (Crutchfield and Young (1989)). In this paper our network is trained to predict video frames; we evaluate it on simple, complex, and noisy videos and show that the final number of neurons is a good indicator of the complexity and predictability of the data stream.