Classification accuracy on the probing tasks for the 8 non-contextualized DSMs.
Task . | PPMI . | SGNS . | GloVe . | FastText . | Majority . | ||||
---|---|---|---|---|---|---|---|---|---|
w2 . | synf . | synt . | w2 . | synf . | synt . | ||||
Positive/Negative | 0.52 | 0.65 | 0.65 | 0.83 | 0.85 | 0.79 | 0.79 | 1.00 | 0.52 |
Concrete/Abstract | 0.69 | 0.70 | 0.73 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.59 |
Animate/Inanimate | 0.74 | 0.78 | 0.86 | 0.94 | 0.96 | 0.97 | 0.96 | 0.98 | 0.83 |
VerbNet | |||||||||
pour-9.5+spray-9.7 | 0.61 | 0.52 | 0.48 | 0.70 | 0.78 | 0.78 | 0.82 | 0.74 | 0.52 |
fill-9.8 | 0.50 | 0.50 | 0.52 | 0.74 | 0.72 | 0.74 | 0.70 | 0.74 | 0.50 |
butter-9.9 | 0.47 | 0.55 | 0.50 | 0.87 | 0.84 | 0.86 | 0.89 | 0.89 | 0.63 |
pocket-9.10 | 0.58 | 0.50 | 0.50 | 0.88 | 0.88 | 0.92 | 0.75 | 0.83 | 0.50 |
remove-10.1+clear-10.3 | 0.44 | 0.56 | 0.44 | 0.56 | 0.68 | 0.76 | 0.68 | 0.64 | 0.52 |
+mine-10.9 | |||||||||
steal-10.5 | 0.48 | 0.48 | 0.41 | 0.83 | 0.83 | 0.79 | 0.86 | 0.90 | 0.52 |
debone-10.8 | 0.59 | 0.63 | 0.54 | 0.86 | 0.82 | 0.82 | 0.68 | 0.90 | 0.50 |
cut-21.1+carve-21.2 | 0.65 | 0.65 | 0.57 | 0.80 | 0.80 | 0.96 | 0.57 | 0.73 | 0.50 |
amalgamate-22.2 | 0.56 | 0.52 | 0.52 | 0.74 | 0.70 | 0.78 | 0.74 | 0.83 | 0.52 |
tape-22.4 | 0.65 | 0.62 | 0.73 | 0.98 | 0.97 | 0.94 | 0.98 | 1.00 | 0.59 |
characterize-29.2 | 0.57 | 0.57 | 0.57 | 0.81 | 0.81 | 0.86 | 0.76 | 0.81 | 0.52 |
amuse-31.1 | 0.55 | 0.56 | 0.51 | 0.69 | 0.80 | 0.75 | 0.67 | 0.72 | 0.51 |
admire-31.2 | 0.44 | 0.52 | 0.56 | 0.87 | 0.91 | 0.87 | 0.78 | 0.96 | 0.52 |
marvel-31.3 | 0.59 | 0.62 | 0.65 | 0.72 | 0.69 | 0.83 | 0.69 | 0.76 | 0.58 |
judgement-33.1 | 0.71 | 0.66 | 0.66 | 0.77 | 0.80 | 0.80 | 0.77 | 0.80 | 0.54 |
manner_of_speaking-37.3 | 0.71 | 0.79 | 0.48 | 0.79 | 0.86 | 0.90 | 0.90 | 0.86 | 0.50 |
say-37.7 | 0.62 | 0.60 | 0.50 | 0.55 | 0.50 | 0.64 | 0.50 | 0.55 | 0.55 |
animal_sounds-38 | 0.67 | 0.80 | 0.67 | 0.87 | 0.90 | 0.83 | 0.83 | 0.93 | 0.56 |
sound_emission-43.2 | 0.52 | 0.48 | 0.61 | 0.70 | 0.78 | 0.74 | 0.65 | 0.70 | 0.52 |
cooking-45.3 | 0.55 | 0.60 | 0.45 | 0.77 | 0.86 | 0.90 | 0.73 | 0.86 | 0.52 |
other_cos-45.4 | 0.54 | 0.50 | 0.57 | 0.70 | 0.73 | 0.74 | 0.76 | 0.68 | 0.50 |
contiguous_location-47.8 | 0.57 | 0.57 | 0.52 | 0.86 | 0.76 | 0.81 | 0.61 | 0.81 | 0.52 |
run-51.3.2 | 0.52 | 0.48 | 0.56 | 0.80 | 0.84 | 0.80 | 0.75 | 0.80 | 0.56 |
Task . | PPMI . | SGNS . | GloVe . | FastText . | Majority . | ||||
---|---|---|---|---|---|---|---|---|---|
w2 . | synf . | synt . | w2 . | synf . | synt . | ||||
Positive/Negative | 0.52 | 0.65 | 0.65 | 0.83 | 0.85 | 0.79 | 0.79 | 1.00 | 0.52 |
Concrete/Abstract | 0.69 | 0.70 | 0.73 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.59 |
Animate/Inanimate | 0.74 | 0.78 | 0.86 | 0.94 | 0.96 | 0.97 | 0.96 | 0.98 | 0.83 |
VerbNet | |||||||||
pour-9.5+spray-9.7 | 0.61 | 0.52 | 0.48 | 0.70 | 0.78 | 0.78 | 0.82 | 0.74 | 0.52 |
fill-9.8 | 0.50 | 0.50 | 0.52 | 0.74 | 0.72 | 0.74 | 0.70 | 0.74 | 0.50 |
butter-9.9 | 0.47 | 0.55 | 0.50 | 0.87 | 0.84 | 0.86 | 0.89 | 0.89 | 0.63 |
pocket-9.10 | 0.58 | 0.50 | 0.50 | 0.88 | 0.88 | 0.92 | 0.75 | 0.83 | 0.50 |
remove-10.1+clear-10.3 | 0.44 | 0.56 | 0.44 | 0.56 | 0.68 | 0.76 | 0.68 | 0.64 | 0.52 |
+mine-10.9 | |||||||||
steal-10.5 | 0.48 | 0.48 | 0.41 | 0.83 | 0.83 | 0.79 | 0.86 | 0.90 | 0.52 |
debone-10.8 | 0.59 | 0.63 | 0.54 | 0.86 | 0.82 | 0.82 | 0.68 | 0.90 | 0.50 |
cut-21.1+carve-21.2 | 0.65 | 0.65 | 0.57 | 0.80 | 0.80 | 0.96 | 0.57 | 0.73 | 0.50 |
amalgamate-22.2 | 0.56 | 0.52 | 0.52 | 0.74 | 0.70 | 0.78 | 0.74 | 0.83 | 0.52 |
tape-22.4 | 0.65 | 0.62 | 0.73 | 0.98 | 0.97 | 0.94 | 0.98 | 1.00 | 0.59 |
characterize-29.2 | 0.57 | 0.57 | 0.57 | 0.81 | 0.81 | 0.86 | 0.76 | 0.81 | 0.52 |
amuse-31.1 | 0.55 | 0.56 | 0.51 | 0.69 | 0.80 | 0.75 | 0.67 | 0.72 | 0.51 |
admire-31.2 | 0.44 | 0.52 | 0.56 | 0.87 | 0.91 | 0.87 | 0.78 | 0.96 | 0.52 |
marvel-31.3 | 0.59 | 0.62 | 0.65 | 0.72 | 0.69 | 0.83 | 0.69 | 0.76 | 0.58 |
judgement-33.1 | 0.71 | 0.66 | 0.66 | 0.77 | 0.80 | 0.80 | 0.77 | 0.80 | 0.54 |
manner_of_speaking-37.3 | 0.71 | 0.79 | 0.48 | 0.79 | 0.86 | 0.90 | 0.90 | 0.86 | 0.50 |
say-37.7 | 0.62 | 0.60 | 0.50 | 0.55 | 0.50 | 0.64 | 0.50 | 0.55 | 0.55 |
animal_sounds-38 | 0.67 | 0.80 | 0.67 | 0.87 | 0.90 | 0.83 | 0.83 | 0.93 | 0.56 |
sound_emission-43.2 | 0.52 | 0.48 | 0.61 | 0.70 | 0.78 | 0.74 | 0.65 | 0.70 | 0.52 |
cooking-45.3 | 0.55 | 0.60 | 0.45 | 0.77 | 0.86 | 0.90 | 0.73 | 0.86 | 0.52 |
other_cos-45.4 | 0.54 | 0.50 | 0.57 | 0.70 | 0.73 | 0.74 | 0.76 | 0.68 | 0.50 |
contiguous_location-47.8 | 0.57 | 0.57 | 0.52 | 0.86 | 0.76 | 0.81 | 0.61 | 0.81 | 0.52 |
run-51.3.2 | 0.52 | 0.48 | 0.56 | 0.80 | 0.84 | 0.80 | 0.75 | 0.80 | 0.56 |