Text generation has become a critical component in modern AI applications such as chatbots and agentic assistants. To combat hallucinations and provide trustworthy output, retrieval augmented generation (RAG) is the current norm for information-dense tasks. Report Generation is a long-form RAG task with strict attestation requirements that makes it well-suited to explore questions of RAG evaluation and multilingual generation. In this task, a long-form report summarizing the relevant information in a corpus is generated in response to a report request, which consists of a user background and an information need. The generated report should provide proper attribution to the source documents to establish trust.
SHARED TASK - CALL FOR PARTICIPATION
Task: Automatic Report Evaluation
We will provide system-generated reports from 2025 TREC RAGTIME submissions that have been judged by human annotators as the input for the shared task participants. The task is to provide a system ranking based on each report request (long-form query with a description of user background) as well as an overall ranking across all report requests. The submitted rankings will be evaluated on correlation to the ranking derived from human annotations.
Task: Multilingual Report Generation
This task involves generating long-form reports in response to a request using information retrieved from a multilingual corpus. Report requests consist of background information about the user and a statement describing their information need in English. In contrast to other RAG tasks, reports should contain only information that is grounded in the corpus. Generated reports should consist of sentences with citations and will be given a length limit. Reports should be written in the same language as the report request. The corpus consists of four million English, Chinese, Russian, and Arabic documents sampled from Common Crawl News, evenly sampled from 2021 to 2024. The organizers will provide search services accessible through an API in addition to the corpus itself. Submitted reports will be judged automatically based on the Auto-ARGUE framework, which scores reports based on whether nuggets of related information are present and correctly cited in the report. We plan to score reports using a range of LLMs to understand their agreement.
IMPORTANT DATES
- Data release: December 10, 2025
- Task submission deadline: March 5, 2026
- Result announcement: April 28, 2026
- System papers due: May 12, 2026
- Workshop dates: July 2 or 3, 2026 (TBA)