Skip to Main Content
Table 1: 
Ablation studies of the Routing Transformer model on the CIFAR-10 data-set. All the models have a total of 12 attention layers and 8 heads. Routing layers when present are always added at the top of the model. A Routing Transformer model with less than 12 routing attention layers and less than 8 routing heads, has the remaining layers and heads of type local attention. A Random Transformer model has a random attention head in place of the routing attention head. We report the performance in bits/dim on the test set and step times are reported on a TPUv3.
ModelRouting headsRouting LayersAttention windowBits/dimSteps/sec
Transformer 3072 2.983 5.608 
Local Transformer 512 3.009 9.023 
Random Transformer 4 (random) 8 (random) 512 3.076 5.448 
 
Routing Transformer 512 3.005 7.968 
Routing Transformer 512 2.986 7.409 
Routing Transformer 512 2.992 6.682 
 
Routing Transformer 512 2.995 7.379 
Routing Transformer 512 2.975 6.492 
Routing Transformer 512 2.991 5.385 
 
Routing Transformer 512 2.995 6.442 
Routing Transformer 512 2.971 5.140 
Routing Transformer 512 3.190 3.897 
 
Routing Transformer 12 512 2.978 5.685 
Routing Transformer 12 512 2.994 4.349 
Routing Transformer 12 512 3.400 3.062 
 
Routing Transformer 1024 2.975 7.344 
Routing Transformer 1024 2.950 6.440 
Routing Transformer 1024 2.982 5.192 
 
Routing Transformer 1024 2.990 6.389 
Routing Transformer 1024 2.958 5.112 
Routing Transformer 1024 3.003 3.674 
 
Routing Transformer 1024 2.991 5.057 
Routing Transformer 1024 2.983 3.597 
Routing Transformer 1024 3.131 2.329 
 
Routing Transformer 12 1024 2.973 4.151 
Routing Transformer 12 1024 3.005 2.788 
Routing Transformer 12 1024 3.291 1.711 
ModelRouting headsRouting LayersAttention windowBits/dimSteps/sec
Transformer 3072 2.983 5.608 
Local Transformer 512 3.009 9.023 
Random Transformer 4 (random) 8 (random) 512 3.076 5.448 
 
Routing Transformer 512 3.005 7.968 
Routing Transformer 512 2.986 7.409 
Routing Transformer 512 2.992 6.682 
 
Routing Transformer 512 2.995 7.379 
Routing Transformer 512 2.975 6.492 
Routing Transformer 512 2.991 5.385 
 
Routing Transformer 512 2.995 6.442 
Routing Transformer 512 2.971 5.140 
Routing Transformer 512 3.190 3.897 
 
Routing Transformer 12 512 2.978 5.685 
Routing Transformer 12 512 2.994 4.349 
Routing Transformer 12 512 3.400 3.062 
 
Routing Transformer 1024 2.975 7.344 
Routing Transformer 1024 2.950 6.440 
Routing Transformer 1024 2.982 5.192 
 
Routing Transformer 1024 2.990 6.389 
Routing Transformer 1024 2.958 5.112 
Routing Transformer 1024 3.003 3.674 
 
Routing Transformer 1024 2.991 5.057 
Routing Transformer 1024 2.983 3.597 
Routing Transformer 1024 3.131 2.329 
 
Routing Transformer 12 1024 2.973 4.151 
Routing Transformer 12 1024 3.005 2.788 
Routing Transformer 12 1024 3.291 1.711 
Close Modal

or Create an Account

Close Modal
Close Modal