User-oriented evaluation scheme for speech translation systems

The main goal of this paper is to propose automatic schemes for the translation paired comparison method. Using the proposed method, experimental results of evaluations that measure how the speech translation system boosts users' ability in cross-lingual communication are also shown. The original translation paired comparison method was proposed by Sugaya et al. to precisely evaluate a speech translation system's capability. Furthermore, the method gives an objective evaluation result, i.e., a score of the Test of English for International Communication (TOEIC). Because the original method requires tremendous evaluation costs, the automation of this method is an important subject for practical use. In the proposed method, currently available automatic evaluation methods (BLEU and a DP-based method) are applied to automate the translation paired comparison method. The results of experiments using the automatic measures show that the proposed method gives similar evaluation results to the original method.