We are pleased to announce the Call for Participation for SemEval-2027 Task 1: RETECO — A Shared Task on Reasoning-Oriented Retrieval (https://semeval.github.io/SemEval2027/tasks). Registration is now open.
RETECO studies retrieval where topical similarity is not enough: a document can match the topic but describe the wrong time period, or only be relevant once the earlier turns of a conversation are understood. Participants build systems that rank the documents that truly support the information need, and optionally generate grounded answers. The training and development data cover 24 domains and more than 1.6 million documents.
The task consists of two tracks and five sub-tracks:
Track 1: Temporal Grounded Retrieval
Sub-track 1a: Temporal Retrieval — rank documents for a temporally complex query.
Sub-track 1b: Step-wise Retrieval — rank documents for each supplied reasoning step of a query.
Track 2: Conversational Retrieval
Sub-track 2a: Conversational Retrieval — rank passages for the current turn of a multi-turn conversation.
Sub-track 2b: Gold-Passage Generation — generate an answer from the gold passages.
Sub-track 2c: Full Conversational RAG — retrieve passages and generate a grounded answer.
Teams may enter one, several, or all five sub-tracks.
How to participate:
Register your team with the form below (one form per team), download the data, and use the starter kit to run the BM25 baseline and score your system locally. Join the task mailing list for evaluation and submission announcements.
Important Dates:
Training and development data: available now
Evaluation period: starts 10 January 2027, closes no later than 31 January 2027
Paper submission: February 2027
Notification to authors: March 2027
Camera-ready submission: April 2027
Additional Information:
Task website: https://datascienceuibk.github.io/RETECO/
Registration form: https://forms.gle/7sKTneQsWDrkcU6JA
Training and development data: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
Starter kit and baseline: https://github.com/DataScienceUIBK/RETECO
RETECO mailing list: https://groups.google.com/g/semeval-2027-reteco
Contact: abdelrahman.abdallah@uibk.ac.at
Organizers:
Abdelrahman Abdallah, Mohammed Ali, Muhammad Abdul-Mageed, Kevin Duh, and Adam Jatowt.
We look forward to your participation!