Event information detection consists of multiple subsequent steps that could drastically affect the quality of the resulted event database. Thus, we believe one must consider a complete scenario that consists of document and sentence classification as relevant or not, event coreference resolution, event information extraction, and event classification in relation to an event taxonomy, and test the results on a list of events created manually to determine performance of the state-of-the-art on this task.
With this objective in mind, we organize a shared task on socio-political and crisis event detection at the workshop. Although the subtasks form a coherent flow, task participants can focus on one or more of them. Therefore, participants can choose the tasks or subtask(s) they would like to participate in. Participants will have access to all of the data for all tasks and subtasks. Any combination of these resources to achieve high performance for any of the tasks is allowed. For instance, Task 1 data could be used to potentially improve the performance on Task 2 and vice versa. The tasks and subtasks are:
Task 1. Multilingual protest news detection
Subtask 1: Document classification ⇒ Does a news article contain information about a past or ongoing event?
Subtask 2: Sentence classification ⇒ Does a sentence contain information about a past or ongoing event?
Subtask 3: Event sentence coreference identification ⇒ Which event sentences (subtask 2) are about the same event?
Subtask 4: Event extraction ⇒ What is the event trigger and its arguments?
We particularly focus on events that are in the scope of contentious politics and characterized by riots and social movements, i.e., the “repertoire of contention” (Giugni 1998, Tarrow 1994, Tilly 1984), which we name GLOCON Gold in our operationalization (Hürriyetoğlu et al. 2020a). The aim of the shared task is to detect and classify socio-political and crisis event information at document, sentence, cross-sentence, and token levels in a multilingual setting. The detailed description of the subtasks can be found in Hürriyetoğlu et al. (2019, 2020b). The data size for English is increased and data for Portuguese, Spanish, and Hindi are added in this edition.
Task 2: Fine-grained classification of Socio-political events
The objective of this task is to evaluate generalized zero-shot learning event classification approaches to classify short text snippets reporting socio-political events with fine-grained event types using the Armed Conflict Location & Event Data Project (ACLED) event taxonomy, which consists of 25 event subtypes pertaining to political violence, demonstrations (rioting and protesting) and selected non-violent, politically important events. The task is to label text snippets using ACLED types and potentially other types of similar events not covered directly by ACLED (unseen classes). One should keep in mind that the event definitions for task 1 and task 2 are not fully compatible.
Task 3: Discovering Black Lives Matter events in United States
This task is only an evaluation task where the participants of task 1 will have the possibility to evaluate their systems on reproducing a manually curated Black Lives Matter (BLM) related protest event list. Participants will use document collections provided by us to extract place and date of the BLM events. The event definition applied for determining these events is the same as the one facilitated for task 1. Participants may utilize any other data source to improve performance of their submissions.
Please find the detailed description of the tasks, application form, sample data, baseline scripts, and submission formats are on the dedicated repository (https://github.com/emerging-welfare/case-2021-shared-task).
Publication
Participants in the Shared Task are expected to submit a paper to the workshop. Submitting a paper is not mandatory for participating in the Shared Task. Papers must follow the CASE 2021 workshop submission instructions (ACL 2021 style template: https://2021.aclweb.org/calls/papers) and will undergo regular peer review. Their acceptance will not depend on the results obtained in the shared task, but on the quality of the paper. Authors of accepted papers will be informed about the evaluation results of their systems prior to the paper submission deadline (see the important dates).
Important Dates for the Shared Task
Release of training data for Task 1: March 14, 2021, Task 2: already available
Registration deadline: April 30, 2021
Release of test data for all tasks to registered participants: May 4 2021
Submission of system responses: May 8, 2021 (12:00 CET)
Results announced to participants: May 10, 2021
Shared Task Papers Due: May 21, 2021
Notification of Acceptance: May 28, 2021
Camera-ready papers due: June 7, 2021
CASE 2021 Workshop (presentation of the ST results): August 5-6, 2021
All deadlines are 23:59 AoE (anywhere on Earth) and in the year 2021, unless otherwise stated above.