HCOMET Measuring Semantic Preservation in Machine Translation (M.Sc. Thesis)

Summary:  HCOMET  was created to be a more reliable system of measuring translation-quality. The Human COgnitive Metric for Evaluating Translation enables researchers to linguistically quantify improvements in translation quality for the development and tuning of machine translation algorithms.

Abstract: Human ranking of machine translation output is a commonly used method for com- paring different innovations in machine translation research. Theoretically simple, the comparison of multiple translations is, in effect, cognitively complex, requiring judges to balance the weight of different types of translation errors in the context of the whole sentence. This cognitive complexity is made evident through low intra- and inter- annotator agreements, which call into question the reliability of such ranking metrics. HMEANT (Lo and Wu, 2011) attempted to decrease the complexity of ranking by dividing sentences into smaller semantic units whose translation alignments were more objective, rendering the task cognitively simpler. However, HMEANT does not discern how these semantic units are related and relies heavily on language-dependent verb frames – a significant problem for a translation metric. This project defines a new set of human metrics focusing on HCOMET (Human COgnitive Metric for Evaluating Translation). HCOMET, attempting to overcome the limitations of HMEANT, em- ployed a new cognitively-informed annotation scheme (Abend and Rappoport, 2013) and new scoring guidelines. While the inter-annotator agreement did not surpass that of HMEANT, the conceptual framework of HCOMET allows for a much more detailed analysis of semantic adequacy in machine translation. (Download the Thesis)

João’s Role: Author

Marinotti, J.  (2014). Measuring Semantic Preservation in Machine Translation with HCOMET: Human Cognitive Metric for Evaluating Translation  (Master’s Thesis). University of Edinburgh.