On the Evaluation of Terminology Translation Errors in NMT and PB-SMT in the Legal Domain: a Study on the Translation of Arabic Legal Documents into English and French

In the translation process, terminological resources are employed to address translation challenges. Therefore, information regarding terminological equivalence is crucial for making the most appropriate choices in terms of translation equivalence. In the context of machine translation, neural models have indeed significantly advanced the state-of-the-art in recent years. However, they still underperform in domain-specific fields and for underresourced languages. This deficiency is particularly evident in the translation of legal terminology for Arabic, where current machine translation outputs fail to adhere to the contextual, linguistic, cultural, and terminological constraints posed by translating legal terms into Arabic. In this paper, we conduct a comparative qualitative evaluation and a comprehensive error analysis of legal terminology translation in Phrase-Based Statistical Machine Translation and Neural Machine Translation in two language pairs: Arabic-English and Arabic-French. We propose an error typology, taking the translation of legal terminology from Arabic into account. We demonstrate our findings, highlighting the strengths and weaknesses of both approaches in the realm of legal terminology translation for Arabic. Additionally, we introduce a multilingual gold standard dataset developed using our Arabic legal corpus. This dataset serves as a reliable benchmark and reference during the evaluation process to determine the degree of adequacy and fluency of the Phrase-Based Statistical Machine Translation and Neural Machine Translation systems.