Dear all,
We are happy to invite you to participate in the 2024 WMT shared task on Quality Estimation. The details of the WMT24 Quality Estimation Task can be found at: https://www2.statmt.org/wmt24/qe-task.html
New this year:
- We introduce a new language pair (zero-shot): English-Spanish
- Continuing from the previous edition we will also analyse the robustness of submitted QE systems to a set of different phenomena which will span from hallucinations and biases to localized errors, which can significantly impact real-world applications.
- We also introduce a new task, seeking not only to detect but also to correct errors: Quality-aware Automatic Post-Editing! We invite participants to submit systems capable of automatically generating QE predictions for machine-translated text and the corresponding output corrections.
2024 QE Tasks:
Task 1 -- Sentence-level quality estimation
This task follows the same format as last year but with fresh test-sets and a new language pair: English-Spanish.
We will test the following language pairs:
- English to German (MQM)
- English to Spanish (MQM)
- English to Hindi (MQM & DA)
- English to Gujarati (DA)
- English to Telugu (DA)
- English to Tamil (DA)
More details: https://www2.statmt.org/wmt24/qe-subtask1.html
Task 2 -- Fine-grained error span detection
Sequence labelling task: predict the error spans in each translation and the associated error severity: Major or Minor.
We will test the following language pairs:
- English to German (MQM)
- English to Spanish (MQM)
- English to Hindi (MQM)
More details: https://www2.statmt.org/wmt24/qe-subtask2.html
Task 3 -- Quality-aware Automatic Post-editing
We expect submissions of post edits, correcting detected error spans of the original translation. Although the task is focused on quality-informed APE, we also allow participants to submit APE output without QE predictions to understand the impact of their QE system. Submissions w/o QE predictions will also be considered official.
We will test the following language pairs:
- English to Hindi
- English to Tamil
More details: https://www2.statmt.org/wmt24/qe-subtask3.html
Important dates:
- Test sets will be released on July 15th.
- Participants can submit their systems by July 23rd on codalab.
- System paper submissions due by 20th August [aligned with WMT deadlines].
Note: Like last year, we aligned with the General MT and Metrics shared tasks to facilitate cross-submission on the common language pairs: English-German, English-Spanish, and English-Hindi (MQM).
We look forward to your submissions and feel free to contact us for further questions!
Best wishes,
WMT 2024 QE ST organisers