FIRST CALL FOR PARTICIPATION
RETECO: A SemEval-2027 Shared Task on Reasoning-Oriented Retrieval
- Website: https://datascienceuibk.github.io/RETECO/
- Data: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- Code: https://github.com/DataScienceUIBK/RETECO
- Mailing list: https://groups.google.com/g/semeval-2027-reteco
OVERVIEW
Information retrieval systems are commonly evaluated primarily in terms of topical relevance, that is, whether retrieved documents concern the same subject as the query. RETECO extends this setting by focusing on retrieval problems in which relevance must be determined through temporal reasoning, conversational context, and multi-step inference over the information need.
In temporal retrieval, relevant evidence must not only address the topic of the query but also correspond to the appropriate time period, version, or temporal relation. Answering such queries may require reasoning over temporal constraints, changes across time, and dependencies between multiple pieces of evidence. For example, a question concerning how a protocol's fee structure changed between two releases requires identifying evidence associated with the relevant versions and periods, rather than documents that merely discuss fees.
In conversational retrieval, the information need associated with a target turn may depend on preceding dialogue context. A query considered in isolation may therefore be ambiguous or underspecified, while its relevant supporting evidence may become identifiable only through context-dependent inference over information established in earlier turns. This setting requires systems to reason over dialogue history, resolve implicit references and dependencies, and determine which evidence supports the current information need.
RETECO brings these settings together in a shared task designed to evaluate reasoning-oriented retrieval and grounded generation. Across five sub-tracks, the task examines systems' ability to perform temporally grounded retrieval, step-wise evidence retrieval, conversational context modelling, and reasoning over retrieved evidence for answer generation. Participants may enter any subset of the sub-tracks, including a single sub-track.
The task is relevant to researchers working on information retrieval, retrieval-augmented generation, temporal reasoning, conversational search, multi-step retrieval, evidence-grounded reasoning, and the evaluation of reasoning-intensive retrieval and generation systems.
TRACKS AND SUB-TRACKS
Track 1 — Temporal Grounded Retrieval (13 domains)
- Sub-track 1a — Temporal Retrieval
- Given a complex temporal query and a domain-specific corpus, return a ranked list of documents. Relevant evidence must satisfy both topical and temporal constraints, requiring systems to identify information aligned with the time periods, versions, or temporal relations expressed in the query. Official metric: nDCG@10.
- Sub-track 1b — Step-wise Temporal Retrieval
- Complex temporal questions may require multiple intermediate retrieval steps. Given a query and an organizer-provided decomposition, return a ranked list of documents for each step. This sub-track evaluates systems' ability to retrieve evidence that supports individual stages of a multi-step reasoning process. Official metric: step-level nDCG@10.
Track 2 — Reasoning-Intensive Conversational Retrieval (11 domains)
- Sub-track 2a — Conversational Retrieval
- Given a conversation and a target turn, return a ranked list of supporting passages. Relevance may depend on information introduced or resolved in earlier turns, requiring systems to model dialogue context, dependencies, and implicit references. Official metric: nDCG@10.
- Sub-track 2b — Gold-Passage Generation
- Given a conversation turn and organizer-provided gold evidence, generate a grounded response. This sub-track isolates answer generation and reasoning over relevant evidence from the retrieval component. Responses are assessed on five independent dimensions: correctness, completeness, relevance, conversational coherence, and faithfulness.
- Sub-track 2c — Full Conversational RAG
- Perform retrieval and grounded generation end to end: return a ranked list of passages and generate a response using the system's own top-ranked evidence. This setting evaluates the complete reasoning-oriented retrieval and generation pipeline.
Official metric: nDCG@10, with generation quality reported alongside.
DATA
The training and development data are already available, in a single download, with no other resources required: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- 24 domains, each self-contained with its own retrieval corpus
- 2,161,196 documents in total
- 1,730 temporal queries and 3,976 decomposed retrieval steps (Track 1)
- 707 conversations and 2,971 target turns (Track 2)
Every domain is split 70/30 into training and development sets with a fixed, published seed, and gold judgments are released for both splits:
- Track 1 queries: 1,211 train / 519 dev
- Track 1 steps: 2,762 train / 1,214 dev
- Track 2 conversations: 496 train / 211 dev
- Track 2 turns: 2,113 train / 858 dev
The retrieval corpus is never split: systems always search the complete corpus of a domain. A query's decomposed steps stay with their query, and every turn of a conversation stays in the same split, so no dialogue context crosses the boundary.
All data are in English. Corpus and question/answer text is distributed under CC BY-SA 4.0, inherited from its Stack Exchange origin; RETECO's own annotations, split assignments and relevance judgments are CC BY 4.0.
The evaluation set is separate, newly annotated, and will be released without gold judgments during the evaluation window.
EVALUATION
The official retrieval metric is nDCG@10, computed using pytrec_eval and macro-averaged across domains, for Sub-tracks 1a, 1b, 2a, and 2c.
For Sub-tracks 2b and 2c, generation quality is evaluated separately from retrieval effectiveness. Generation scores are reported alongside the retrieval leaderboard rather than combined into a single aggregate score, allowing retrieval and grounded generation performance to remain independently interpretable.
In addition to the official metric, the task reports diagnostic measures intended to characterize different aspects of system behaviour and reasoning performance. These include temporal precision, temporal relevance, temporal coverage, per-turn-depth breakdowns, MAP@10, Recall@10, MRR, and per-domain results.
BASELINES
A starter kit is available with a BM25 baseline, the official scorer, and a submission format checker: https://github.com/DataScienceUIBK/RETECO/tree/main/starter_kit
BM25 reference results on the released splits (nDCG@10, macro-averaged over domains):
- Sub-track 1a, whole query: 0.0879 train / 0.0967 dev
- Sub-track 1b, query plus step instruction: 0.0852 train / 0.1063 dev
- Sub-track 2a, current turn only: 0.1837 train / 0.1827 dev
- Sub-track 2a, turn plus conversation history: 0.4539 train / 0.4379 dev
The baseline results illustrate two central challenges addressed by RETECO. For Track 1, the comparatively low BM25 scores indicate that lexical overlap alone provides limited effectiveness for queries requiring temporal alignment and multi-step reasoning. This leaves substantial scope for approaches that explicitly model temporal constraints, query decomposition, and evidence dependencies.
For Track 2, retrieval effectiveness changes substantially when conversation history is incorporated into the query representation. Using the same BM25 retriever, adding dialogue context increases nDCG@10 from 0.1827 to 0.4379 on the development set. This result highlights the importance of contextual reasoning and dialogue-state representation in identifying relevant evidence for conversational information needs.
IMPORTANT DATES
- 8 August 2026 — Sample data released
- 30 August 2026 — Training and development data released
- 10 January 2027 — Evaluation window opens
- 31 January 2027 — Evaluation window closes (at the latest)
- February 2027 — System description papers due (tentative)
- March 2027 — Notification to authors (tentative)
- April 2027 — Camera-ready papers due (tentative)
- Summer 2027 — SemEval-2027 workshop
Paper and workshop dates follow the central SemEval-2027 calendar and remain tentative. The exact RETECO closing date will be announced and will be no later than 31 January 2027.
HOW TO PARTICIPATE
- Join the mailing list to receive announcements: https://groups.google.com/g/semeval-2027-reteco
- Download the data: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- Build a system. Develop against the training split and keep the development split as a held-out check. The starter kit reproduces the baseline numbers above and scores runs with the same metric the leaderboard uses.
- Submit predictions for the held-out evaluation set during the evaluation window. The competition platform, registration form, and submission limits will be announced on the website and to the mailing list.
Participating teams will be invited to submit system description papers to the SemEval-2027 workshop proceedings.
Full participation guide: https://datascienceuibk.github.io/RETECO/participate.html
ORGANIZERS
- Abdelrahman Abdallah — University of Innsbruck
- Mohammed Ali — University of Innsbruck
- Muhammad Abdul-Mageed — University of British Columbia
- Kevin Duh — Johns Hopkins University
- Adam Jatowt — University of Innsbruck
CONTACT
- Mailing list: https://groups.google.com/g/semeval-2027-reteco
- Email: semeval-2027-reteco [at] googlegroups.com
- Email: abdelrahman.abdallah [at] uibk.ac.at