Skip to Main Content
Table 3:
Penn Treebank Character-Level Modeling Test on GORU, GRU, LSTM, and EURNN.
Model bpc Number of Units 
LSTM 1.656 184 
GRU 1.639 216 
EURNN 1.747 1024 
uRNN 1.750 1024 
oRNN (Mhammedi et al., 2017) 1.68 512 (m=510
GORU (EURNN FFT-style) 1.654 256 
GORU (Householder) 1.652 256 
GRU (w/ modReLU) 1.702 216 
GORU (w/ ReLU) 1.785 256 
GORU (w/ tanh) 1.780 256 
GORU (w/o reset gate) 1.759 256 
GORU (w/o update gate) 1.718 256 
Model bpc Number of Units 
LSTM 1.656 184 
GRU 1.639 216 
EURNN 1.747 1024 
uRNN 1.750 1024 
oRNN (Mhammedi et al., 2017) 1.68 512 (m=510
GORU (EURNN FFT-style) 1.654 256 
GORU (Householder) 1.652 256 
GRU (w/ modReLU) 1.702 216 
GORU (w/ ReLU) 1.785 256 
GORU (w/ tanh) 1.780 256 
GORU (w/o reset gate) 1.759 256 
GORU (w/o update gate) 1.718 256 

Notes: We use only single-layer models. We choose the size of the models to match the number of parameters, which is about 184,000 for each model. GORU is able to outperform unitary and orthogonal RNNs. We also tested the performance of restricted GORU, which shows the necessity of both reset and update gates.

Close Modal

or Create an Account

Close Modal
Close Modal