New Research Confirms ModernMT Outperforms Leading MT and GenAI

After Gartner, ID, and CSA Research positioned ModernMT as the leading machine translation system for enterprise needs, a new independent report by Achim Ruopp, which also includes GPT-4, confirms in extensive technical detail that ModernMT is the best-performing MT solution for enterprises.

Achim Ruopp of Polyglot Technology evaluated Translated's adaptive MT solution against public MT systems and found that ModernMT performed the best across the board.

Evaluations and comparisons were performed using publicly available datasets and algorithms based on commonly used evaluation metrics (COMET, TER, and SacreBLEU).

GPT-4 was also included in the evaluations, which showed that ModernMT continued to outperform even state-of-the-art LLMs for translations.

Rome – February 8, 2024

Translated is proud to share that an independent study led by Achim Ruopp, founder of Polyglot Technology, has demonstrated how ModernMT's adaptive machine translation (MT) solution can outperform major publicly available MT systems, including DeepL and Google Translate, in real use cases. The research used publicly available algorithms and datasets, making evaluation and comparison transparent and reproducible.

The Translated team also ran OpenAI's GPT-4, the state-of-the-art large language model, through the same tests and quality evaluation and found that GPT-4 consistently performed worse than any of the other leading neural MT services tested.

This benchmark shows that ModernMT, with adaptive MT and document context, works extremely well for typical language industry translation projects, especially if translation organizations have large, diverse sets of existing translation data. All this is possible without the effort of training and maintaining customized MT systems.
Achim Ruopp – Polyglot Technology's founder

Until now, companies looking for the ideal MT solution have been limited to third-party evaluations focused primarily on comparing public MT systems' generic (or static) versions. This approach overlooks the unique capability of ModernMT's adaptive model, which comes out of the box adapted to the user's specific content. As a result, these reports provide a suboptimal view of the full range of MT solutions available.

The comparison conducted by Polyglot Technology highlights the significant impact of ModernMT's instant adaptation compared to generic (or static) MT systems.

Polyglot Technology's data further validates ModernMT's leadership in machine translation as recognized by other independent reports such as IDC MarketScape, CSA Research, and Gartner.

About the Research

Polyglot Technology’s research employed commonly used metrics (COMET, TER, and SacreBLEU) and tested the leading public MT systems (Amazon Translate, DeepL Translator, Google Translate, and Microsoft Translator) against ModernMT models (static, adaptive, adaptive with an external TM of 10k segments). The study was based on publicly available evaluations and comparison scripts to translate a public US English dataset from Autodesk into German, Italian, Spanish, Brazilian Portuguese, and Simplified Chinese. It focused on the MT systems' ability to handle different languages, contexts, and specialized terminology, providing a practical and direct comparison of these tools in typical translation workflows.

This approach allows companies to test the leading public MT solutions with their content.

Translated commissioned Polyglot Technology to create a publicly available, transparent, and easily reproducible evaluation and comparison toolset to encourage organizations to evaluate first-hand, with their data, the benefits of adaptive MT over static models. Our team is available to provide support and guidance in running the tests.

This study further demonstrates that ModernMT's adaptive capabilities are a game-changer in the industry, providing an unparalleled level of accuracy and context awareness out of the box that static models simply can't match without additional effort.
Disheng Qiu – VP of Engineering at Translated

Download the report

Read the detailed evaluation report to better understand the research methodology and process. The details will help the reader to better understand the out-of-the-box performance of ModernMT against leading generic public NMT systems, and to better evaluate and compare MT systems for their unique requirements.