Dependency SRL Results with pre-identified predicates (w/ pred) and without pre-identified predicates (w/o pred) on the CoNLL-2009 English in-domain (WSJ) and out-of-domain (Brown) test sets. The “PLM” column indicates whether and which pre-trained language model is used, the “SYN” column indicates whether syntax information is employed, and “+E” in the “PLM” column shows that the model leverages ELMo for pre-trained language model features. [Ens.] is used to specify the ensemble system, [Semi.] indicates semi-supervised training is adopted, and [Joint] means joint learning with other tasks.
System . | PLM . | SYN . | w/ pred . | w/o pred . | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
WSJ . | Brown . | WSJ . | Brown . | |||||||||||
. | P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | ||
(Zhao et al. 2009b) | Y | − | − | 85.4 | − | − | 73.3 | − | − | − | − | − | − | |
(Zhao et al. 2009a) | Y | − | − | 86.2 | − | − | 74.6 | − | − | − | − | − | − | |
(Lei et al. 2015) | Y | − | − | 86.6 | − | − | 75.6 | − | − | − | − | − | − | |
(FitzGerald et al. 2015) | Y | − | − | 87.3 | − | − | 75.2 | − | − | − | − | − | − | |
[Ens.] (FitzGerald et al. 2015) | Y | − | − | 87.8 | − | − | 75.5 | − | − | − | − | − | − | |
(Roth and Lapata 2016) | Y | 90.0 | 85.5 | 87.7 | 78.6 | 73.8 | 76.1 | − | − | − | − | − | − | |
[Ens.] (Roth and Lapata 2016) | Y | 90.3 | 85.7 | 87.9 | 79.7 | 73.6 | 76.5 | − | − | − | − | − | − | |
(Swayamdipta et al. 2016) | N | − | − | 85.0 | − | − | − | − | − | 80.5 | − | − | − | |
(Marcheggiani and Titov 2017) | Y | 89.1 | 86.8 | 88.0 | 78.5 | 75.9 | 77.2 | − | − | − | − | − | − | |
[Ens.] (Marcheggiani and Titov 2017) | Y | 90.5 | 87.7 | 89.1 | 80.8 | 77.1 | 78.9 | − | − | − | − | − | − | |
(Marcheggiani, Frolov, and Titov 2017) | N | 88.7 | 86.8 | 87.7 | 79.4 | 76.2 | 77.7 | − | − | − | − | − | − | |
(Mulcaire, Swayamdipta, and Smith 2018) | N | − | − | 87.2 | − | − | − | − | − | − | − | − | − | |
(Kasai et al. 2019) | Y | 89.0 | 88.2 | 88.6 | 78.0 | 77.2 | 77.6 | − | − | − | − | − | − | |
+E | Y | 90.3 | 90.0 | 90.2 | 81.0 | 80.5 | 80.8 | − | − | − | − | − | − | |
(Cai and Lapata 2019a) | N | 91.1 | 90.4 | 90.7 | 82.1 | 81.3 | 81.6 | − | − | − | − | − | − | |
[Semi.] (Cai and Lapata 2019a) | N | 91.7 | 90.8 | 91.2 | 83.2 | 81.9 | 82.5 | − | − | − | − | − | − | |
(Zhang, Wang, and Si 2019) | Y | 89.6 | 86.0 | 87.7 | − | − | − | − | − | − | − | − | − | |
(Lyu, Cohen, and Titov 2019) | +E | N | − | − | 91.0 | − | − | 82.2 | − | − | − | − | − | − |
(Chen, Lyu, and Titov 2019) | +E | N | 90.7 | 91.4 | 91.1 | 82.7 | 82.8 | 82.7 | − | − | − | − | − | − |
[Joint] (Zhou, Li, and Zhao 2020) | N | 88.7 | 89.8 | 89.3 | 82.5 | 83.2 | 82.8 | 84.2 | 87.6 | 85.9 | 76.5 | 78.5 | 77.5 | |
+E | N | 89.7 | 90.9 | 90.3 | 83.9 | 85.0 | 84.5 | 85.2 | 88.2 | 86.7 | 78.6 | 80.8 | 79.7 | |
Sequence-based (2018b; 2018) | +E | N | 89.5 | 87.9 | 88.7 | 81.7 | 76.1 | 78.8 | 83.5 | 82.4 | 82.9 | 71.5 | 70.9 | 71.2 |
+K-order Hard Pruning (2018b) | +E | Y | 89.7 | 89.3 | 89.5 | 81.9 | 76.9 | 79.3 | 83.9 | 82.7 | 83.3 | 71.5 | 71.3 | 71.4 |
+SynRule Soft Pruning | +E | Y | 89.9 | 89.1 | 89.5 | 78.8 | 81.2 | 80.0 | 82.9 | 84.3 | 83.6 | 70.9 | 72.1 | 71.5 |
+GCN Syntax Encoder (2018) | +E | Y | 90.3 | 89.3 | 89.8 | 80.6 | 79.0 | 79.8 | 85.3 | 82.5 | 83.9 | 71.9 | 71.5 | 71.7 |
+SA-LSTM Syntax Encoder (2018) | +E | Y | 90.8 | 88.6 | 89.7 | 81.0 | 78.2 | 79.6 | 85.3 | 82.6 | 84.0 | 71.8 | 71.6 | 71.7 |
+Tree-LSTM Syntax Encoder (2018) | +E | Y | 90.0 | 88.8 | 89.4 | 80.4 | 78.7 | 79.5 | 83.1 | 83.7 | 83.4 | 70.9 | 72.1 | 71.5 |
Tree-based (2018) | +E | N | 89.2 | 90.4 | 89.8 | 80.0 | 78.6 | 79.3 | 84.8 | 85.4 | 85.1 | 72.4 | 74.0 | 73.2 |
+K-order Hard Pruning | +E | Y | 90.3 | 89.5 | 89.9 | 80.0 | 79.0 | 79.5 | 83.9 | 86.5 | 85.2 | 73.6 | 72.8 | 73.2 |
+SynRule Soft Pruning (2019) | +E | Y | 90.0 | 90.7 | 90.3 | 79.6 | 80.4 | 80.0 | 84.9 | 85.9 | 85.4 | 72.7 | 74.3 | 73.5 |
+GCN Syntax Encoder | +E | Y | 90.9 | 90.1 | 90.5 | 81.4 | 78.8 | 80.1 | 86.1 | 84.9 | 85.5 | 73.5 | 73.7 | 73.6 |
+SA-LSTM Syntax Encoder | +E | Y | 91.1 | 89.9 | 90.5 | 80.9 | 79.5 | 80.2 | 85.3 | 85.0 | 85.2 | 72.9 | 73.5 | 73.2 |
+Tree-LSTM Syntax Encoder | +E | Y | 89.8 | 90.6 | 90.2 | 80.0 | 79.8 | 79.9 | 85.3 | 85.3 | 85.3 | 73.9 | 73.1 | 73.5 |
Graph-based (2019a) | +E | N | 89.6 | 91.2 | 90.4 | 81.7 | 81.4 | 81.5 | 85.6 | 85.0 | 85.3 | 73.0 | 74.0 | 73.5 |
+K-order Hard Pruning | +E | Y | 90.3 | 89.7 | 90.0 | 80.7 | 81.9 | 81.3 | 84.6 | 85.8 | 85.2 | 73.7 | 73.3 | 73.5 |
+SynRule Soft Pruning | +E | Y | 89.8 | 90.6 | 90.2 | 80.8 | 82.4 | 81.6 | 85.0 | 86.0 | 85.5 | 72.8 | 74.4 | 73.6 |
+GCN Syntax Encoder | +E | Y | 90.5 | 91.7 | 91.1 | 83.3 | 80.9 | 82.1 | 86.2 | 86.0 | 86.1 | 73.8 | 74.6 | 74.2 |
+SA-LSTM Syntax Encoder | +E | Y | 91.0 | 90.4 | 90.7 | 82.4 | 81.6 | 82.0 | 86.3 | 85.5 | 85.9 | 75.4 | 72.8 | 74.1 |
+Tree-LSTM Syntax Encoder | +E | Y | 90.7 | 90.3 | 90.5 | 80.2 | 83.4 | 81.8 | 86.9 | 84.3 | 85.6 | 74.1 | 73.7 | 73.9 |
System . | PLM . | SYN . | w/ pred . | w/o pred . | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
WSJ . | Brown . | WSJ . | Brown . | |||||||||||
. | P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | ||
(Zhao et al. 2009b) | Y | − | − | 85.4 | − | − | 73.3 | − | − | − | − | − | − | |
(Zhao et al. 2009a) | Y | − | − | 86.2 | − | − | 74.6 | − | − | − | − | − | − | |
(Lei et al. 2015) | Y | − | − | 86.6 | − | − | 75.6 | − | − | − | − | − | − | |
(FitzGerald et al. 2015) | Y | − | − | 87.3 | − | − | 75.2 | − | − | − | − | − | − | |
[Ens.] (FitzGerald et al. 2015) | Y | − | − | 87.8 | − | − | 75.5 | − | − | − | − | − | − | |
(Roth and Lapata 2016) | Y | 90.0 | 85.5 | 87.7 | 78.6 | 73.8 | 76.1 | − | − | − | − | − | − | |
[Ens.] (Roth and Lapata 2016) | Y | 90.3 | 85.7 | 87.9 | 79.7 | 73.6 | 76.5 | − | − | − | − | − | − | |
(Swayamdipta et al. 2016) | N | − | − | 85.0 | − | − | − | − | − | 80.5 | − | − | − | |
(Marcheggiani and Titov 2017) | Y | 89.1 | 86.8 | 88.0 | 78.5 | 75.9 | 77.2 | − | − | − | − | − | − | |
[Ens.] (Marcheggiani and Titov 2017) | Y | 90.5 | 87.7 | 89.1 | 80.8 | 77.1 | 78.9 | − | − | − | − | − | − | |
(Marcheggiani, Frolov, and Titov 2017) | N | 88.7 | 86.8 | 87.7 | 79.4 | 76.2 | 77.7 | − | − | − | − | − | − | |
(Mulcaire, Swayamdipta, and Smith 2018) | N | − | − | 87.2 | − | − | − | − | − | − | − | − | − | |
(Kasai et al. 2019) | Y | 89.0 | 88.2 | 88.6 | 78.0 | 77.2 | 77.6 | − | − | − | − | − | − | |
+E | Y | 90.3 | 90.0 | 90.2 | 81.0 | 80.5 | 80.8 | − | − | − | − | − | − | |
(Cai and Lapata 2019a) | N | 91.1 | 90.4 | 90.7 | 82.1 | 81.3 | 81.6 | − | − | − | − | − | − | |
[Semi.] (Cai and Lapata 2019a) | N | 91.7 | 90.8 | 91.2 | 83.2 | 81.9 | 82.5 | − | − | − | − | − | − | |
(Zhang, Wang, and Si 2019) | Y | 89.6 | 86.0 | 87.7 | − | − | − | − | − | − | − | − | − | |
(Lyu, Cohen, and Titov 2019) | +E | N | − | − | 91.0 | − | − | 82.2 | − | − | − | − | − | − |
(Chen, Lyu, and Titov 2019) | +E | N | 90.7 | 91.4 | 91.1 | 82.7 | 82.8 | 82.7 | − | − | − | − | − | − |
[Joint] (Zhou, Li, and Zhao 2020) | N | 88.7 | 89.8 | 89.3 | 82.5 | 83.2 | 82.8 | 84.2 | 87.6 | 85.9 | 76.5 | 78.5 | 77.5 | |
+E | N | 89.7 | 90.9 | 90.3 | 83.9 | 85.0 | 84.5 | 85.2 | 88.2 | 86.7 | 78.6 | 80.8 | 79.7 | |
Sequence-based (2018b; 2018) | +E | N | 89.5 | 87.9 | 88.7 | 81.7 | 76.1 | 78.8 | 83.5 | 82.4 | 82.9 | 71.5 | 70.9 | 71.2 |
+K-order Hard Pruning (2018b) | +E | Y | 89.7 | 89.3 | 89.5 | 81.9 | 76.9 | 79.3 | 83.9 | 82.7 | 83.3 | 71.5 | 71.3 | 71.4 |
+SynRule Soft Pruning | +E | Y | 89.9 | 89.1 | 89.5 | 78.8 | 81.2 | 80.0 | 82.9 | 84.3 | 83.6 | 70.9 | 72.1 | 71.5 |
+GCN Syntax Encoder (2018) | +E | Y | 90.3 | 89.3 | 89.8 | 80.6 | 79.0 | 79.8 | 85.3 | 82.5 | 83.9 | 71.9 | 71.5 | 71.7 |
+SA-LSTM Syntax Encoder (2018) | +E | Y | 90.8 | 88.6 | 89.7 | 81.0 | 78.2 | 79.6 | 85.3 | 82.6 | 84.0 | 71.8 | 71.6 | 71.7 |
+Tree-LSTM Syntax Encoder (2018) | +E | Y | 90.0 | 88.8 | 89.4 | 80.4 | 78.7 | 79.5 | 83.1 | 83.7 | 83.4 | 70.9 | 72.1 | 71.5 |
Tree-based (2018) | +E | N | 89.2 | 90.4 | 89.8 | 80.0 | 78.6 | 79.3 | 84.8 | 85.4 | 85.1 | 72.4 | 74.0 | 73.2 |
+K-order Hard Pruning | +E | Y | 90.3 | 89.5 | 89.9 | 80.0 | 79.0 | 79.5 | 83.9 | 86.5 | 85.2 | 73.6 | 72.8 | 73.2 |
+SynRule Soft Pruning (2019) | +E | Y | 90.0 | 90.7 | 90.3 | 79.6 | 80.4 | 80.0 | 84.9 | 85.9 | 85.4 | 72.7 | 74.3 | 73.5 |
+GCN Syntax Encoder | +E | Y | 90.9 | 90.1 | 90.5 | 81.4 | 78.8 | 80.1 | 86.1 | 84.9 | 85.5 | 73.5 | 73.7 | 73.6 |
+SA-LSTM Syntax Encoder | +E | Y | 91.1 | 89.9 | 90.5 | 80.9 | 79.5 | 80.2 | 85.3 | 85.0 | 85.2 | 72.9 | 73.5 | 73.2 |
+Tree-LSTM Syntax Encoder | +E | Y | 89.8 | 90.6 | 90.2 | 80.0 | 79.8 | 79.9 | 85.3 | 85.3 | 85.3 | 73.9 | 73.1 | 73.5 |
Graph-based (2019a) | +E | N | 89.6 | 91.2 | 90.4 | 81.7 | 81.4 | 81.5 | 85.6 | 85.0 | 85.3 | 73.0 | 74.0 | 73.5 |
+K-order Hard Pruning | +E | Y | 90.3 | 89.7 | 90.0 | 80.7 | 81.9 | 81.3 | 84.6 | 85.8 | 85.2 | 73.7 | 73.3 | 73.5 |
+SynRule Soft Pruning | +E | Y | 89.8 | 90.6 | 90.2 | 80.8 | 82.4 | 81.6 | 85.0 | 86.0 | 85.5 | 72.8 | 74.4 | 73.6 |
+GCN Syntax Encoder | +E | Y | 90.5 | 91.7 | 91.1 | 83.3 | 80.9 | 82.1 | 86.2 | 86.0 | 86.1 | 73.8 | 74.6 | 74.2 |
+SA-LSTM Syntax Encoder | +E | Y | 91.0 | 90.4 | 90.7 | 82.4 | 81.6 | 82.0 | 86.3 | 85.5 | 85.9 | 75.4 | 72.8 | 74.1 |
+Tree-LSTM Syntax Encoder | +E | Y | 90.7 | 90.3 | 90.5 | 80.2 | 83.4 | 81.8 | 86.9 | 84.3 | 85.6 | 74.1 | 73.7 | 73.9 |