It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Standard

It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. / Bugliarello, Emanuele; Mielke, Sabrina J.; Anastasopoulos, Antonios; Cotterell, Ryan; Okazaki, Naoaki.

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online : Association for Computational Linguistics (ACL), 2020. p. 1640-1649.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Harvard

Bugliarello, E, Mielke, SJ, Anastasopoulos, A, Cotterell, R & Okazaki, N 2020, It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL), Online, pp. 1640-1649, 58th Annual Meeting of the Association for Computational Linguistics, Online, 05/07/2020. https://doi.org/10.18653/v1/2020.acl-main.149

APA

Bugliarello, E., Mielke, S. J., Anastasopoulos, A., Cotterell, R., & Okazaki, N. (2020). It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 1640-1649). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.149

Vancouver

Bugliarello E, Mielke SJ, Anastasopoulos A, Cotterell R, Okazaki N. It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics (ACL). 2020. p. 1640-1649 https://doi.org/10.18653/v1/2020.acl-main.149

Author

Bugliarello, Emanuele ; Mielke, Sabrina J. ; Anastasopoulos, Antonios ; Cotterell, Ryan ; Okazaki, Naoaki. / It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online : Association for Computational Linguistics (ACL), 2020. pp. 1640-1649

Bibtex

@inproceedings{6826b84a768e400caa3ddc07a046953a,
title = "It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information",
abstract = "The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.",
author = "Emanuele Bugliarello and Mielke, {Sabrina J.} and Antonios Anastasopoulos and Ryan Cotterell and Naoaki Okazaki",
year = "2020",
month = jul,
day = "1",
doi = "10.18653/v1/2020.acl-main.149",
language = "English",
pages = "1640--1649",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
publisher = "Association for Computational Linguistics (ACL)",
address = "United States",
note = "58th Annual Meeting of the Association for Computational Linguistics ; Conference date: 05-07-2020 Through 10-07-2020",

}

RIS

TY - GEN

T1 - It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

AU - Bugliarello, Emanuele

AU - Mielke, Sabrina J.

AU - Anastasopoulos, Antonios

AU - Cotterell, Ryan

AU - Okazaki, Naoaki

PY - 2020/7/1

Y1 - 2020/7/1

N2 - The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.

AB - The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.

U2 - 10.18653/v1/2020.acl-main.149

DO - 10.18653/v1/2020.acl-main.149

M3 - Article in proceedings

SP - 1640

EP - 1649

BT - Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

CY - Online

T2 - 58th Annual Meeting of the Association for Computational Linguistics

Y2 - 5 July 2020 through 10 July 2020

ER -

ID: 255126547