Ablation study results of the proposed models for each hardness level on dev set. SPC, SCN, and JTF represent the use of Statement Position Code, supplemented column names, and JOIN table filtering, respectively.
| Approaches . | SPC . | SCN . | JTF . | Easy . | Med. . | Hard . | Extra . | ALL . |
|---|---|---|---|---|---|---|---|---|
| RYANSQL | O | O | O | 69.2% | 43.0% | 28.2% | 22.4% | 43.4% |
| O | O | X | 68.0% | 40.2% | 27.6% | 19.4% | 41.4% | |
| O | X | O | 62.0% | 38.2% | 24.1% | 14.7% | 37.7% | |
| O | X | X | 65.2% | 37.7% | 29.9% | 18.8% | 39.9% | |
| X | O | O | 63.2% | 39.5% | 19.0% | 16.5% | 38.0% | |
| X | O | X | 68.4% | 41.6% | 18.4% | 14.7% | 39.7% | |
| X | X | O | 63.2% | 39.3% | 14.9% | 16.5% | 37.2% | |
| X | X | X | 60.0% | 38.2% | 16.7% | 11.8% | 35.5% | |
| RYANSQL + BERT | O | O | O | 86.0% | 70.5% | 54.6% | 40.6% | 66.6% |
| O | O | X | 86.8% | 66.1% | 46.6% | 42.4% | 63.9% | |
| O | X | O | 76.4% | 58.2% | 46.6% | 30.6% | 56.1% | |
| O | X | X | 78.0% | 63.4% | 46.0% | 28.8% | 58.3% | |
| X | O | O | 85.6% | 66.6% | 27.0% | 22.4% | 57.3% | |
| X | O | X | 83.6% | 68.4% | 25.9% | 26.5% | 58.0% | |
| X | X | O | 78.4% | 60.2% | 21.3% | 24.7% | 52.2% | |
| X | X | X | 77.2% | 60.5% | 23.6% | 25.9% | 52.6% | |
| Approaches . | SPC . | SCN . | JTF . | Easy . | Med. . | Hard . | Extra . | ALL . |
|---|---|---|---|---|---|---|---|---|
| RYANSQL | O | O | O | 69.2% | 43.0% | 28.2% | 22.4% | 43.4% |
| O | O | X | 68.0% | 40.2% | 27.6% | 19.4% | 41.4% | |
| O | X | O | 62.0% | 38.2% | 24.1% | 14.7% | 37.7% | |
| O | X | X | 65.2% | 37.7% | 29.9% | 18.8% | 39.9% | |
| X | O | O | 63.2% | 39.5% | 19.0% | 16.5% | 38.0% | |
| X | O | X | 68.4% | 41.6% | 18.4% | 14.7% | 39.7% | |
| X | X | O | 63.2% | 39.3% | 14.9% | 16.5% | 37.2% | |
| X | X | X | 60.0% | 38.2% | 16.7% | 11.8% | 35.5% | |
| RYANSQL + BERT | O | O | O | 86.0% | 70.5% | 54.6% | 40.6% | 66.6% |
| O | O | X | 86.8% | 66.1% | 46.6% | 42.4% | 63.9% | |
| O | X | O | 76.4% | 58.2% | 46.6% | 30.6% | 56.1% | |
| O | X | X | 78.0% | 63.4% | 46.0% | 28.8% | 58.3% | |
| X | O | O | 85.6% | 66.6% | 27.0% | 22.4% | 57.3% | |
| X | O | X | 83.6% | 68.4% | 25.9% | 26.5% | 58.0% | |
| X | X | O | 78.4% | 60.2% | 21.3% | 24.7% | 52.2% | |
| X | X | X | 77.2% | 60.5% | 23.6% | 25.9% | 52.6% | |