Measurements at the single cell level showed that monoclonal Escherichia coli cells differ widely in the numbers of components affecting gene expression dynamics. Using a stochastic model of a 2-genes symmetric toggle switch with realistic multi-step promoter initiation kinetics and empirically validated parameter values, we investigate the role of transcription initiation kinetics on the degree with which cell-to-cell variability in cellular components generates cell-to-cell diversity in switch dynamics. We find that while the mean switching frequency is determined by the promoter kinetics, the cell to cell diversity of this frequency depends both on promoter kinetics and diversity in RNA polymerase numbers. At a microscale level, the main regulator of the cell-to-cell variability in protein numbers (of both genes in ON and OFF states) is the promoters kinetics, not the diversity in RNA polymerase numbers. We conclude that the promoters kinetics is a critical regulator of the toggle switch dynamics and that can be used as a regulatable filter of extrinsic noise.