An example of the CCG supertag predictions for the verb “use” from four different BERT variants. The correct answer is “((S[b]∖NP)/PP)/NP”, which both the R2L-KD and UG-KD predict correctly (blue). However, the No-KD baseline and the L2R-KD model produce (the same) incorrect predictions (red); both models fail to subcategorize for the prepositional phrase “as screens” as a dependent of the verb “use”. Beyond this, all four models predict the correct supertags for all other words (not shown).