CfP: Shared Task on Text-Graph Representations for KGQA @ TextGraphs-17
Venue: ACL 2024, TextGraph workshop
Website: https://sites.google.com/view/textgraphs2024/home/shared-task
TextGraph-17 workshop, co-located with ACL-2024, features a shared task. TextGraph fosters the investigation of synergies between methods for text and graph processing. This edition focuses on the fusion of LLMs with KGs. In line with this goal, we propose a shared task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).
The shared task is to select a KG entity (out of several candidates) that corresponds to an answer given a textual question. The specificity of the task is that for each question-answer (Q-A) pair, not only a textual Q-A pair is provided but also a graph of shortest paths in the KG from entities in the query to the LLM-generated candidate entity (including links of the intermediate nodes). This way, participants may easily experiment with various strategies of text-graph modality fusion for the given task in a controllable manner.
Participants can submit reports about their participation in the TextGraphs workshop. An example of a previous TextGraphs shared task can be found here: https://aclanthology.org/volumes/2022.textgraphs-1/.
Task Description
Participants are given:
text: question with a list of Wikidata entities mentioned,
text: 5-10 answer candidates in the form of Wikidata entities,
graph: a Wikidata sub-graph composed of the shortest paths between entities in question and entities in answer is provided.
One of the candidates is correct, and the others are incorrect. The goal is to find the correct answer, i.e., perform a binary classification. Participants are provided a train and development dataset in the form of Q-A-Subgraph triples. Besides, a submission to Codalab's public and private leaderboard of the test set Q-A solutions will be available.
Important Dates
Training dataset released: 10th March 2024
Test set released: 25 March 2024
End of evaluation: 25 April 2024
Submission deadline for technical reports: 17 May 2024
Contact
Please write all questions about the shared task to textgraphs17 [at] googlegroups.com. Also, you are invited to join our Telegram group, where you can connect to organizers and get updates: https://t.me/+kRTCZYTrpJ5jZGVi
Organizers:
Irina Nikishina, Universität Hamburg
Aida Usmanova, Leuphana University Lüneburg
Angelie Kraft, Universität Hamburg
Cedric Möller, Universität Hamburg
Debayan Banerjee, Universität Hamburg
Junbo Huang, Universität Hamburg
Longquan Jiang, Universität Hamburg
Rana Abdullah, Universität Hamburg
Xi Yan, Universität Hamburg
Andrey Sakhovskiy, KFU
Elena Tutubalina, KFU
Mikhail Salnikov, AIRI
Alexander Panchenko, AIRI
Ricardo Usbeck, Universität Hamburg