We study the subthreshold voltage fluctuations of a conductance-based passive point neuron stimulated by filtered Poissonian shot noise. We give exact analytical expressions in terms of quadratures for the first two time-dependent moments and the autocorrelation function of the membrane voltage. We also derive simplified expressions for the moments in terms of elementary functions that hold true in the limit case of short filter time, small spike amplitude, and a single synaptic reversal potential. By means of these expressions, we show that for an ensemble of equilibrated conductances but sharp initial voltage (corresponding to a short voltage clamp at the initial time), the mean and the standard deviation can display nonmonotonic time courses. In particular, transient changes in the standard deviation disagree strongly with the predictions of the commonly used effective time constant approximation over a large parameter range. We also study the dependence of the correlation time of the voltage on the synaptic spike amplitude and the synaptic input rate. All results are confirmed by extensive stochastic simulations.