Dependency SRL results on the CoNLL-2009 multilingual in-domain test sets with pre-identified predicates (w/ pred) setting. The first row is the best result of the CoNLL-2009 shared task (Hajič et al. 2009). The “PLM” column indicates whether and which pre-trained language model is used, the “SYN” column indicates whether syntax information is employed, “+E” in the “PLM” column indicates the model leverages pre-trained ELMo features for all languages, and “+E†” in the “PLM” column indicates ELMo is only used for English.
| System . | PLM . | SYN . | CA . | CS . | DE . | EN . | ES . | JA . | ZH . | Avg. . |
|---|---|---|---|---|---|---|---|---|---|---|
| CoNLL-2009 best | Y | 80.3 | 86.5 | 79.7 | 86.2 | 80.5 | 78.3 | 78.6 | 81.4 | |
| (Zhao et al. 2009a) | Y | 80.3 | 85.2 | 76.0 | 85.4 | 80.5 | 78.2 | 77.7 | 80.5 | |
| (Roth and Lapata 2016) | Y | − | − | 80.1 | 87.7 | 80.2 | − | 79.4 | − | |
| (Marcheggiani and Titov 2017) | Y | − | − | − | 88.0 | − | − | 82.5 | − | |
| (Marcheggiani, Frolov, and Titov 2017) | N | − | 86.0 | − | 87.7 | 80.3 | − | 81.2 | − | |
| (Mulcaire, Swayamdipta, and Smith 2018) | N | 79.5 | 85.1 | 70.0 | 87.2 | 77.3 | 76.0 | 81.9 | 79.6 | |
| (Kasai et al. 2019) | +E | Y | − | − | − | 90.2 | 83.0 | − | − | |
| (Cai and Lapata 2019a) | N | − | − | 83.3 | 90.7 | 82.1 | − | 84.6 | − | |
| [Semi.] (Cai and Lapata 2019a) | N | − | − | 83.8 | 91.2 | 82.9 | − | 85.0 | − | |
| (Zhang, Wang, and Si 2019) | Y | − | − | − | 87.7 | − | − | 84.2 | − | |
| (Lyu, Cohen, and Titov 2019) | +E† | N | 80.9 | 87.6 | 75.9 | 91.0 | 80.5 | 82.5 | 83.3 | 83.1 |
| (Chen, Lyu, and Titov 2019) | +E† | N | 81.7 | 88.1 | 76.4 | 91.1 | 81.3 | 81.3 | 81.7 | 83.1 |
| Sequence-based (2018b; 2018) | +E | N | 84.0 | 87.8 | 76.8 | 88.7 | 82.9 | 82.8 | 83.1 | 83.7 |
| +K-order Hard Pruning (2018b) | +E | Y | 84.5 | 88.3 | 77.3 | 89.5 | 83.3 | 82.9 | 82.8 | 84.1 |
| +SynRule Soft Pruning | +E | Y | 84.4 | 88.2 | 77.5 | 89.5 | 83.2 | 83.0 | 83.3 | 84.2 |
| +GCN Syntax Encoder (2018) | +E | Y | 84.6 | 88.5 | 77.2 | 89.8 | 83.6 | 83.2 | 83.8 | 84.4 |
| +SA-LSTM Syntax Encoder (2018) | +E | Y | 84.3 | 88.5 | 77.0 | 89.7 | 83.5 | 83.1 | 83.5 | 84.2 |
| +Tree-LSTM Syntax Encoder (2018) | +E | Y | 84.1 | 88.3 | 76.9 | 89.4 | 83.2 | 82.9 | 83.4 | 84.0 |
| Tree-based (2018) | +E | N | 84.1 | 88.4 | 78.4 | 89.9 | 83.5 | 83.0 | 84.0 | 84.5 |
| +K-order Hard Pruning | +E | Y | 84.2 | 88.5 | 78.4 | 89.9 | 83.4 | 82.8 | 84.2 | 84.5 |
| +SynRule Soft Pruning (2019) | +E | Y | 84.4 | 88.8 | 78.5 | 90.0 | 83.7 | 83.1 | 84.6 | 84.7 |
| +GCN Syntax Encoder | +E | Y | 84.8 | 89.3 | 78.1 | 90.2 | 84.0 | 83.3 | 85.0 | 85.0 |
| +SA-LSTM Syntax Encoder | +E | Y | 84.6 | 89.0 | 78.8 | 90.0 | 83.7 | 83.1 | 84.8 | 84.9 |
| +Tree-LSTM Syntax Encoder | +E | Y | 84.4 | 88.9 | 78.6 | 89.9 | 83.6 | 83.0 | 84.5 | 84.7 |
| Graph-based (2019a) | +E | N | 85.0 | 90.2 | 76.0 | 90.0 | 83.8 | 82.7 | 85.7 | 84.8 |
| +K-order Hard Pruning | +E | Y | 84.9 | 90.2 | 75.7 | 89.8 | 83.5 | 82.8 | 85.8 | 84.7 |
| +SynRule Soft Pruning | +E | Y | 85.2 | 90.3 | 76.2 | 90.1 | 84.0 | 82.9 | 85.8 | 84.9 |
| +GCN Syntax Encoder | +E | Y | 85.5 | 90.5 | 76.6 | 90.4 | 84.3 | 83.2 | 86.1 | 85.2 |
| +SA-LSTM Syntax Encoder | +E | Y | 85.2 | 90.5 | 76.4 | 90.3 | 84.1 | 83.2 | 86.0 | 85.1 |
| +Tree-LSTM Syntax Encoder | +E | Y | 85.0 | 90.3 | 76.2 | 90.3 | 84.0 | 83.0 | 85.8 | 84.9 |
| System . | PLM . | SYN . | CA . | CS . | DE . | EN . | ES . | JA . | ZH . | Avg. . |
|---|---|---|---|---|---|---|---|---|---|---|
| CoNLL-2009 best | Y | 80.3 | 86.5 | 79.7 | 86.2 | 80.5 | 78.3 | 78.6 | 81.4 | |
| (Zhao et al. 2009a) | Y | 80.3 | 85.2 | 76.0 | 85.4 | 80.5 | 78.2 | 77.7 | 80.5 | |
| (Roth and Lapata 2016) | Y | − | − | 80.1 | 87.7 | 80.2 | − | 79.4 | − | |
| (Marcheggiani and Titov 2017) | Y | − | − | − | 88.0 | − | − | 82.5 | − | |
| (Marcheggiani, Frolov, and Titov 2017) | N | − | 86.0 | − | 87.7 | 80.3 | − | 81.2 | − | |
| (Mulcaire, Swayamdipta, and Smith 2018) | N | 79.5 | 85.1 | 70.0 | 87.2 | 77.3 | 76.0 | 81.9 | 79.6 | |
| (Kasai et al. 2019) | +E | Y | − | − | − | 90.2 | 83.0 | − | − | |
| (Cai and Lapata 2019a) | N | − | − | 83.3 | 90.7 | 82.1 | − | 84.6 | − | |
| [Semi.] (Cai and Lapata 2019a) | N | − | − | 83.8 | 91.2 | 82.9 | − | 85.0 | − | |
| (Zhang, Wang, and Si 2019) | Y | − | − | − | 87.7 | − | − | 84.2 | − | |
| (Lyu, Cohen, and Titov 2019) | +E† | N | 80.9 | 87.6 | 75.9 | 91.0 | 80.5 | 82.5 | 83.3 | 83.1 |
| (Chen, Lyu, and Titov 2019) | +E† | N | 81.7 | 88.1 | 76.4 | 91.1 | 81.3 | 81.3 | 81.7 | 83.1 |
| Sequence-based (2018b; 2018) | +E | N | 84.0 | 87.8 | 76.8 | 88.7 | 82.9 | 82.8 | 83.1 | 83.7 |
| +K-order Hard Pruning (2018b) | +E | Y | 84.5 | 88.3 | 77.3 | 89.5 | 83.3 | 82.9 | 82.8 | 84.1 |
| +SynRule Soft Pruning | +E | Y | 84.4 | 88.2 | 77.5 | 89.5 | 83.2 | 83.0 | 83.3 | 84.2 |
| +GCN Syntax Encoder (2018) | +E | Y | 84.6 | 88.5 | 77.2 | 89.8 | 83.6 | 83.2 | 83.8 | 84.4 |
| +SA-LSTM Syntax Encoder (2018) | +E | Y | 84.3 | 88.5 | 77.0 | 89.7 | 83.5 | 83.1 | 83.5 | 84.2 |
| +Tree-LSTM Syntax Encoder (2018) | +E | Y | 84.1 | 88.3 | 76.9 | 89.4 | 83.2 | 82.9 | 83.4 | 84.0 |
| Tree-based (2018) | +E | N | 84.1 | 88.4 | 78.4 | 89.9 | 83.5 | 83.0 | 84.0 | 84.5 |
| +K-order Hard Pruning | +E | Y | 84.2 | 88.5 | 78.4 | 89.9 | 83.4 | 82.8 | 84.2 | 84.5 |
| +SynRule Soft Pruning (2019) | +E | Y | 84.4 | 88.8 | 78.5 | 90.0 | 83.7 | 83.1 | 84.6 | 84.7 |
| +GCN Syntax Encoder | +E | Y | 84.8 | 89.3 | 78.1 | 90.2 | 84.0 | 83.3 | 85.0 | 85.0 |
| +SA-LSTM Syntax Encoder | +E | Y | 84.6 | 89.0 | 78.8 | 90.0 | 83.7 | 83.1 | 84.8 | 84.9 |
| +Tree-LSTM Syntax Encoder | +E | Y | 84.4 | 88.9 | 78.6 | 89.9 | 83.6 | 83.0 | 84.5 | 84.7 |
| Graph-based (2019a) | +E | N | 85.0 | 90.2 | 76.0 | 90.0 | 83.8 | 82.7 | 85.7 | 84.8 |
| +K-order Hard Pruning | +E | Y | 84.9 | 90.2 | 75.7 | 89.8 | 83.5 | 82.8 | 85.8 | 84.7 |
| +SynRule Soft Pruning | +E | Y | 85.2 | 90.3 | 76.2 | 90.1 | 84.0 | 82.9 | 85.8 | 84.9 |
| +GCN Syntax Encoder | +E | Y | 85.5 | 90.5 | 76.6 | 90.4 | 84.3 | 83.2 | 86.1 | 85.2 |
| +SA-LSTM Syntax Encoder | +E | Y | 85.2 | 90.5 | 76.4 | 90.3 | 84.1 | 83.2 | 86.0 | 85.1 |
| +Tree-LSTM Syntax Encoder | +E | Y | 85.0 | 90.3 | 76.2 | 90.3 | 84.0 | 83.0 | 85.8 | 84.9 |