FIRST CALL FOR PARTICIPATION
RETECO: A SemEval-2027 Shared Task on Reasoning-Oriented Retrieval
- Website: https://datascienceuibk.github.io/RETECO/
- Data: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- Code: https://github.com/DataScienceUIBK/RETECO
- Mailing list: https://groups.google.com/g/semeval-2027-reteco
The training and development data are already released. So are the baselines, the official scorer, and a submission format checker. You can build and evaluate a system today.
OVERVIEW
Retrieval systems are usually judged on topical relevance: does this document discuss the same subject as the query? RETECO asks a harder question. Can a system find the evidence that matters because of when it applies, and because of what a conversation has already established?
Both are ordinary requirements in practice and both defeat topical matching. A question about how a protocol's fee structure changed between two releases needs evidence anchored to particular periods, not merely documents about fees. A question asked as the fourth turn of a conversation may be unintelligible in isolation, and the passage that answers it may be relevant only given what was resolved three turns earlier.
RETECO brings these two settings together in one shared task with five sub-tracks. Participants may enter any subset, including a single sub-track.
The task is relevant to researchers working on information retrieval, retrieval-augmented generation, temporal reasoning, conversational search, and the evaluation of reasoning-intensive systems.
TRACKS AND SUB-TRACKS
Track 1 — Temporal Grounded Retrieval (13 domains)
- Sub-track 1a — Temporal Retrieval
- Given a complex temporal query and a domain corpus, return a ranked list of documents. Evidence must be topically relevant and temporally aligned with the information need. Official metric: nDCG@10.
- Sub-track 1b — Step-wise Temporal Retrieval
- Complex temporal questions decompose into intermediate retrieval steps. Given a query and its supplied decomposition, return a ranked list per step. Official metric: step-level nDCG@10.
Track 2 — Reasoning-Intensive Conversational Retrieval (11 domains)
- Sub-track 2a — Conversational Retrieval
- Given a conversation and a target turn, return a ranked list of supporting passages. Relevance depends on the dialogue context, not the turn alone. Official metric: nDCG@10.
- Sub-track 2b — Gold-Passage Generation
- Given a conversation turn and organizer-provided gold evidence, generate a grounded response. Assessed on five independent dimensions: correctness, completeness, relevance, conversational coherence, and faithfulness.
- Sub-track 2c — Full Conversational RAG
- Retrieve and generate end to end: return a ranking, then produce a grounded answer from your own top retrieved passages. Official metric: nDCG@10, with generation quality reported alongside.
DATA
The training and development data are available now, in a single download, with no other resources required: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- 24 domains, each self-contained with its own retrieval corpus
- 2,161,196 documents in total
- 1,730 temporal queries and 3,976 decomposed retrieval steps (Track 1)
- 707 conversations and 2,971 target turns (Track 2)
Every domain is split 70/30 into training and development sets with a fixed, published seed, and gold judgments are released for both splits:
- Track 1 queries: 1,211 train / 519 dev
- Track 1 steps: 2,762 train / 1,214 dev
- Track 2 conversations: 496 train / 211 dev
- Track 2 turns: 2,113 train / 858 dev
The retrieval corpus is never split: systems always search the complete corpus of a domain. A query's decomposed steps stay with their query, and every turn of a conversation stays in the same split, so no dialogue context crosses the boundary.
All data are in English. Corpus and question/answer text is distributed under CC BY-SA 4.0, inherited from its Stack Exchange origin; RETECO's own annotations, split assignments and relevance judgments are CC BY 4.0.
The evaluation set is separate, newly annotated, and will be released without gold judgments during the evaluation window.
EVALUATION
The official retrieval metric is nDCG@10, computed with pytrec_eval and macro-averaged, for Sub-tracks 1a, 1b, 2a and 2c. Generation quality for 2b and 2c is reported alongside the retrieval leaderboard rather than combined into it, so the ranking stays interpretable.
Diagnostics reported in addition to the official metric include temporal precision, temporal relevance, temporal coverage, per-turn-depth breakdowns, MAP@10, Recall@10, MRR, and per-domain results.
BASELINES
A starter kit is available with a BM25 baseline, the official scorer, and a submission format checker: https://github.com/DataScienceUIBK/RETECO/tree/main/starter_kit
BM25 reference results on the released splits, nDCG@10, macro-averaged over domains:
- Sub-track 1a, whole query: 0.0879 train / 0.0967 dev
- Sub-track 1b, query plus step instruction: 0.0852 train / 0.1063 dev
- Sub-track 2a, current turn only: 0.1837 train / 0.1827 dev
- Sub-track 2a, turn plus conversation history: 0.4539 train / 0.4379 dev
Two things are visible in these numbers. Lexical matching is weak on Track 1: temporal grounding is not a keyword problem, and the headroom is substantial. On Track 2 the query representation dominates — appending the conversation history more than doubles the same retriever — which is precisely the modelling question the track is designed to expose.
IMPORTANT DATES
- 8 August 2026 — Sample data released
- 30 August 2026 — Training and development data released
- 10 January 2027 — Evaluation window opens
- 31 January 2027 — Evaluation window closes (at the latest)
- February 2027 — System description papers due (tentative)
- March 2027 — Notification to authors (tentative)
- April 2027 — Camera-ready papers due (tentative)
- Summer 2027 — SemEval-2027 workshop
Paper and workshop dates follow the central SemEval-2027 calendar and remain tentative. The exact RETECO closing date will be announced and will be no later than 31 January 2027.
HOW TO PARTICIPATE
- Join the mailing list to receive announcements: https://groups.google.com/g/semeval-2027-reteco
- Download the data: https://huggingface.co/datasets/DataScience-UIBK/RETECO-SemEval2027
- Build a system. Develop against the training split and keep the development split as a held-out check. The starter kit reproduces the baseline numbers above and scores runs with the same metric the leaderboard uses.
- Submit predictions for the held-out evaluation set during the evaluation window. The competition platform, registration form, and submission limits will be announced on the website and to the mailing list.
Participating teams will be invited to submit system description papers to the SemEval-2027 workshop proceedings.
Full participation guide: https://datascienceuibk.github.io/RETECO/participate.html
ORGANIZERS
- Abdelrahman Abdallah — University of Innsbruck
- Mohammed Ali — University of Innsbruck
- Muhammad Abdul-Mageed — University of British Columbia
- Kevin Duh — Johns Hopkins University
- Adam Jatowt — University of Innsbruck
CONTACT
- Mailing list: https://groups.google.com/g/semeval-2027-reteco
- Email: semeval-2027-reteco [at] googlegroups.com
- Email: abdelrahman.abdallah [at] uibk.ac.at