Overview and categorization of papers dealing with position information. We categorize along two dimensions: a keyword and topic, which describes the main topic of a paper, and whose reference point is used for the position encodings.
| . | Reference Point . | |||
|---|---|---|---|---|
| Absolute . | Absolute & Relative . | Relative . | ||
| Topic | Generic | Devlin et al. (2019) | Shaw, Uszkoreit, and Vaswani (2018) | Dai et al. (2019) |
| Kitaev, Kaiser, and Levskaya (2020) | Ke, He, and Liu (2021) | Raffel et al. (2020) | ||
| Liu et al. (2020) | Dufter, Schmitt, and Schútze (2020) | Chang et al. (2021) | ||
| Press, Smith, and Lewis (2021) | He et al. (2021) | Wu, Wu, and Huang (2021) | ||
| Wang et al. (2020) | Huang et al. (2020) | |||
| Shen et al. (2018) | ||||
| Neishi and Yoshinaga (2019) | ||||
| Liutkus et al. (2021) | ||||
| Sinusoidal | Vaswani et al. (2017) | Yan et al. (2019) | ||
| Dehghani et al. (2019) | Su et al. (2021) | |||
| Li et al. (2019) | ||||
| Likhomanenko et al. (2021) | ||||
| Graphs | Shiv and Quirk (2019) | Wang et al. (2019) | Zhu et al. (2019) | |
| Dwivedi and Bresson (2020) | Zhang et al. (2020) | Cai and Lam (2020) | ||
| Schmitt et al. (2021) | ||||
| Decoder | Takase and Okazaki (2019) | |||
| Oka et al. (2020) | ||||
| Bao et al. (2019) | ||||
| Crossling. | Artetxe, Ruder, and Yogatama (2020) | |||
| Ding, Wang, and Tao (2020) | ||||
| Liu et al. (2021a) | ||||
| Liu et al. (2021b) | ||||
| Analysis | Yang et al. (2019) | Rosendahl et al. (2019) | ||
| Wang and Chen (2020) | Wang et al. (2021) | |||
| Chen et al. (2021) | ||||
| . | Reference Point . | |||
|---|---|---|---|---|
| Absolute . | Absolute & Relative . | Relative . | ||
| Topic | Generic | Devlin et al. (2019) | Shaw, Uszkoreit, and Vaswani (2018) | Dai et al. (2019) |
| Kitaev, Kaiser, and Levskaya (2020) | Ke, He, and Liu (2021) | Raffel et al. (2020) | ||
| Liu et al. (2020) | Dufter, Schmitt, and Schútze (2020) | Chang et al. (2021) | ||
| Press, Smith, and Lewis (2021) | He et al. (2021) | Wu, Wu, and Huang (2021) | ||
| Wang et al. (2020) | Huang et al. (2020) | |||
| Shen et al. (2018) | ||||
| Neishi and Yoshinaga (2019) | ||||
| Liutkus et al. (2021) | ||||
| Sinusoidal | Vaswani et al. (2017) | Yan et al. (2019) | ||
| Dehghani et al. (2019) | Su et al. (2021) | |||
| Li et al. (2019) | ||||
| Likhomanenko et al. (2021) | ||||
| Graphs | Shiv and Quirk (2019) | Wang et al. (2019) | Zhu et al. (2019) | |
| Dwivedi and Bresson (2020) | Zhang et al. (2020) | Cai and Lam (2020) | ||
| Schmitt et al. (2021) | ||||
| Decoder | Takase and Okazaki (2019) | |||
| Oka et al. (2020) | ||||
| Bao et al. (2019) | ||||
| Crossling. | Artetxe, Ruder, and Yogatama (2020) | |||
| Ding, Wang, and Tao (2020) | ||||
| Liu et al. (2021a) | ||||
| Liu et al. (2021b) | ||||
| Analysis | Yang et al. (2019) | Rosendahl et al. (2019) | ||
| Wang and Chen (2020) | Wang et al. (2021) | |||
| Chen et al. (2021) | ||||