MIT Press

Table 1:

Ablation studies of the Routing Transformer model on the CIFAR-10 data-set. All the models have a total of 12 attention layers and 8 heads. Routing layers when present are always added at the top of the model. A Routing Transformer model with less than 12 routing attention layers and less than 8 routing heads, has the remaining layers and heads of type local attention. A Random Transformer model has a random attention head in place of the routing attention head. We report the performance in bits/dim on the test set and step times are reported on a TPUv3.

Model	Routing heads	Routing Layers	Attention window	Bits/dim	Steps/sec
Transformer	0	0	3072	2.983	5.608
Local Transformer	0	0	512	3.009	9.023
Random Transformer	4 (random)	8 (random)	512	3.076	5.448

Routing Transformer	2	2	512	3.005	7.968
Routing Transformer	4	2	512	2.986	7.409
Routing Transformer	8	2	512	2.992	6.682

Routing Transformer	2	4	512	2.995	7.379
Routing Transformer	4	4	512	2.975	6.492
Routing Transformer	8	4	512	2.991	5.385

Routing Transformer	2	8	512	2.995	6.442
Routing Transformer	4	8	512	2.971	5.140
Routing Transformer	8	8	512	3.190	3.897

Routing Transformer	2	12	512	2.978	5.685
Routing Transformer	4	12	512	2.994	4.349
Routing Transformer	8	12	512	3.400	3.062

Routing Transformer	2	2	1024	2.975	7.344
Routing Transformer	4	2	1024	2.950	6.440
Routing Transformer	8	2	1024	2.982	5.192

Routing Transformer	2	4	1024	2.990	6.389
Routing Transformer	4	4	1024	2.958	5.112
Routing Transformer	8	4	1024	3.003	3.674

Routing Transformer	2	8	1024	2.991	5.057
Routing Transformer	4	8	1024	2.983	3.597
Routing Transformer	8	8	1024	3.131	2.329

Routing Transformer	2	12	1024	2.973	4.151
Routing Transformer	4	12	1024	3.005	2.788
Routing Transformer	8	12	1024	3.291	1.711

Model	Routing heads	Routing Layers	Attention window	Bits/dim	Steps/sec
Transformer	0	0	3072	2.983	5.608
Local Transformer	0	0	512	3.009	9.023
Random Transformer	4 (random)	8 (random)	512	3.076	5.448

Routing Transformer	2	2	512	3.005	7.968
Routing Transformer	4	2	512	2.986	7.409
Routing Transformer	8	2	512	2.992	6.682

Routing Transformer	2	4	512	2.995	7.379
Routing Transformer	4	4	512	2.975	6.492
Routing Transformer	8	4	512	2.991	5.385

Routing Transformer	2	8	512	2.995	6.442
Routing Transformer	4	8	512	2.971	5.140
Routing Transformer	8	8	512	3.190	3.897

Routing Transformer	2	12	512	2.978	5.685
Routing Transformer	4	12	512	2.994	4.349
Routing Transformer	8	12	512	3.400	3.062

Routing Transformer	2	2	1024	2.975	7.344
Routing Transformer	4	2	1024	2.950	6.440
Routing Transformer	8	2	1024	2.982	5.192

Routing Transformer	2	4	1024	2.990	6.389
Routing Transformer	4	4	1024	2.958	5.112
Routing Transformer	8	4	1024	3.003	3.674

Routing Transformer	2	8	1024	2.991	5.057
Routing Transformer	4	8	1024	2.983	3.597
Routing Transformer	8	8	1024	3.131	2.329

Routing Transformer	2	12	1024	2.973	4.151
Routing Transformer	4	12	1024	3.005	2.788
Routing Transformer	8	12	1024	3.291	1.711

View Large

This Feature Is Available To Subscribers Only