Skip to Main Content

Besides reporting the Bleu scores, we look at the “readability” of translations, similar to the study conducted by Charniak, Knight, and Yamada (2003). The translations are sorted into four groups: good/bad syntax crossed with good/bad meaning by human judges (see Table 12). We find that many more sentences are perfect, many more are grammatically correct, and many more are semantically correct. The syntactic language model (Charniak et al. 2003) only improves translations to have good grammar, but does not improve translations to preserve meaning. The composite 5-gram/2-SLM + 2-gram/4-SLM + 5-gram/PLSA1 language model improves both significantly. Bear in mind that Charniak et al. (2003) integrated Charniak's language model with the syntax-based translation model proposed by Yamada and Knight (2001) to rescore a tree-to-string translation forest, whereas we use only our language model for N-best list re-ranking. Also, the same study (Charniak et al. 2003) found that the outputs produced using the n-grams received higher scores from Bleu; ours did not. The difference between human judgments and Bleu scores indicates that closer agreement may be possible by incorporating syntactic structure and semantic information into the Bleu score evaluation. For example, semantically similar words like insure and ensure as in Bleu paper (Papineni et al. 2002) should be substituted in the formula, and there is a weight to measure the goodness of syntactic structure. This modification will lead to a better metric and such information can be provided by our composite language models.

Table 12 

Results of “readability” evaluation on 919 translated sentences of 100 documents. P = perfect; S = only semantically correct; G = only grammatically correct; W = wrong.

system model
P
S
G
W
Baseline 95 398 20 406 
5-gram 122 406 24 367 
5-gram/2-SLM + 2-gram/4-SLM + 5-gram/PLSA1 151 425 33 310 
system model
P
S
G
W
Baseline 95 398 20 406 
5-gram 122 406 24 367 
5-gram/2-SLM + 2-gram/4-SLM + 5-gram/PLSA1 151 425 33 310 

Close Modal

or Create an Account

Close Modal
Close Modal