Many continuous-time term structure of interest rate models assume a factor structure where the drift and volatility functions are affine functions of the state-variable process. These models involve very specific parametric choices of factors and functional specifications of the drift and volatility. Moreover, under the affine term structure restrictions not all factors necessarily affect interest rates at all maturities simultaneously. This class of so-called affine models covers a wide variety of existing empirical as well as theoretical models in the literature. In this paper we take a very agnostic approach to the specification of these diffusion functions and test implications of the affine term structure restrictions. We do not test a specific model among the class of affine models per se. Instead, the affine term structure restrictions we test are based on the derivatives of the responses of interest rates to the factors. We also test how many and which factors affect a particular rate. These tests are conducted within a framework which models interest rates as functions of “fundamental” factors, and the responses of interest rates to these factors are estimated with nonparametric methods. We consider two sets of factors, one based on key macroeconomic variables, and one based on interest rate spreads. In general, despite their common use we find that the empirical evidence does not support the restrictions imposed by affine models. Besides testing the affine structure restrictions we also uncover a set of fundamental factors which appear remarkably robust in explaining interest rate dynamics at the long and short maturities we consider.