We investigate rhythms in networks of neurons with recurrent excitation, that is, with excitatory cells exciting each other. Recurrent excitation can sustain activity even when the cells in the network are driven below threshold, too weak to fire on their own. This sort of “reverberating” activity is often thought to be the basis of working memory. Recurrent excitation can also lead to “runaway” transitions, sudden transitions to high-frequency firing; this may be related to epileptic seizures. Not all fundamental questions about these phenomena have been answered with clarity in the literature. We focus on three questions here: (1) How much recurrent excitation is needed to sustain reverberating activity? How does the answer depend on parameters? (2) Is there a positive minimum frequency of reverberating activity, a positive “onset frequency”? How does it depend on parameters? (3) When do runaway transitions occur? For reduced models, we give mathematical answers to these questions. We also examine computationally to which extent our findings are reflected in the behavior of biophysically more realistic model networks. Our main results can be summarized as follows. (1) Reverberating activity can be fueled by extremely weak slow recurrent excitation, but only by sufficiently strong fast recurrent excitation. (2) The onset of reverberating activity, as recurrent excitation is strengthened or external drive is raised, occurs at a positive frequency. It is faster when the external drive is weaker (and the recurrent excitation stronger). It is slower when the recurrent excitation has a longer decay time constant. (3) Runaway transitions occur only with fast, not with slow, recurrent excitation. We also demonstrate that the relation between reverberating activity fueled by recurrent excitation and runaway transitions can be visualized in an instructive way by a (generalized) cusp catastrophe surface.