Comparison according to several criteria: = (Absolute, Relative, or Both); = Injection method (ape or mam); = Are the position representations learned during training?; = Is position information recurring at each layer vs. only before first layer?; = Can the position model generalize to longer inputs than a fixed value?;# Param = Number of parameters introduced by the position model (notation follows the text paper, d hidden dimension, h # attention heads, n vocabulary size, t_max longest sequence length, l # layers). The - symbol means that an entry does not fit into our categories. Note that a model as a whole can combine different position models while this comparison focuses on the respective novel part(s).