Neural oscillations provide a means for efficient and flexible communication among different brain areas. Understanding the mechanisms of the generation of brain oscillations is crucial to determine principles of communication and information transfer in the brain circuits. It is well known that the inhibitory neurons play a major role in the generation of oscillations in the gamma range, in pure inhibitory networks, or in the networks composed of excitatory and inhibitory neurons. In this study, we explore the impact of different parameters and, in particular, the delay in the transmission of the signals between the neurons, on the dynamics of inhibitory networks. We show that increasing delay in a reasonable range increases the synchrony and stabilizes the oscillations. Unstable gamma oscillations characterized by a highly variable amplitude of oscillations can be observed in an intermediate range of delays. We show that in this range of delays, other experimentally observed phenomena such as sparse firing, variable amplitude and period, and the correlation between the instantaneous amplitude and period could be observed. The results broaden our understanding of the mechanism of the generation of the gamma oscillations in the inhibitory networks, known as the ING (interneuron-gamma) mechanism.