Abstract
We study the asymptotic properties of the sequence of iterates of weight-vector estimates obtained by training a feedforward neural network with a basic gradient-descent method using a fixed learning rate and no batch-processing. Earlier results based on stochastic approximation techniques (Kuan and Hornik 1991; Finnoff 1993; Bucklew et al. 1993) have established the existence of a gaussian limiting distribution for the weights, but they apply only in the limiting case of a zero learning rate. We here prove, from an exact analysis of the one-dimensional case and constant learning rate, weak convergence to a distribution that is not gaussian in general. We also run simulations to compare and contrast the results of our analysis with those of stochastic approximation.