To understand how biological and bio-inspired complex computational networks can function in the presence of noise and damage, we have evolved very small spiking neural networks in the presence of noise on the membrane potential. The networks were built with adaptive exponential integrate and fire neurons. The simple but not trivial task we evolved the networks for consisted of recognizing a short temporal pattern in the activity of the network inputs. This task can be described in abstract terms as finding a specific subsequence of symbols (“ABC”) in a continuous sequence of symbols (“..ABCCCAAABCAC..”). We show that networks with three interneurons and one output neuron can solve this task in the presence of biologically plausible levels of noise. We describe how such a network works by mapping its activity onto the state of a finite state transducer—an abstract model of computation on continuous time series. We demonstrate that the networks evolved with noise are much more robust than networks evolved without noise to the modification of neuronal parameters and variation of the properties of the input. We also show that the networks evolved with noise are denser and have stronger connections than the networks evolved without noise. Finally, we demonstrate the emergence of memory in the evolved networks—sustained spiking of some neurons maintained thanks to the presence of self-excitatory loops.