Call for participation: WASSA-2017 Shared Task on Emotion Intensity
Part of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA-2017), which is to be held in conjunction with EMNLP-2017.
Task webpage: http://saifmohammad.com/WebPages/EmotionIntensity-SharedTask.html
Task mailing group: EmotionIntensity [at] googlegroups.com
Background and Significance:
Existing emotion datasets are mainly annotated categorically without an indication of degree of emotion. Further, the tasks are almost always framed as classification tasks (identify 1 among n emotions for this sentence). In contrast, it is often useful for applications to know the degree to which an emotion is expressed in text. In this task, systems have to automatically determine the intensity of emotions in tweets.
Task:
Given a tweet and an emotion X, determine the intensity or degree of emotion X felt by the speaker -- a real-valued score between 0 and 1. The maximum possible score 1 stands for feeling the maximum amount of emotion X (or having a mental state maximally inclined towards feeling emotion X). The minimum possible score 0 stands for feeling the least amount of emotion X (or having a mental state maximally away from feeling emotion X). The tweet along with the emotion X will be referred to as an instance. Note that the absolute scores have no inherent meaning -- they are used only as a means to convey that the instances with higher scores correspond to a greater degree of emotion X than instances with lower scores.
Data:
Training, development, and test datasets are provided for four emotions: joy, sadness, fear, and anger. For example, the anger training dataset has tweets along with a real-valued score between 0 and 1 indicating the degree of anger felt by the speaker. The test data includes only the tweet text. Gold emotion intensity scores will be released after the evaluation period.
Evaluation:
For each emotion, systems are evaluated by calculating the Pearson Correlation Coefficient with Gold ratings. The correlation scores across all four emotions will be averaged to determine the bottom-line competition metric by which the submissions will be ranked.
The official evaluation script (which also acts as a format checker) is available for download. You may want to run it on the training set to determine your progress, and eventually on the test set to check the format of your submission.
Web Hosting of the Competition:
The entire competition will be hosted on CodaLab Competitions (https://competitions.codalab.org/). A direct link to the Emotion Intensity CodaLab competition is here: https://competitions.codalab.org/competitions/16380
(CodaLab has been used in many research evaluation competitions in the past such as Microsoft COCO Image Captioning Challenge and SemEval-2017.)
Paper:
Participants will be given the opportunity to write a system-description paper that describes their system, resources used, results, and analysis. This paper will be part of the official WASSA-2017 proceedings. The paper is to be four pages long plus two pages at most for references. The papers are to follow the format and style files provided by EMNLP-2017.
Schedule:
Training data ready: Data for anger, fear, and joy are already available; data for sadness will be made available in the second half of February 2017
Evaluation period starts: May 02, 2017
Evaluation period ends: May 14, 2017
Results posted: May 21, 2017
Workshop paper submission deadline: June 10, 2017
Author notifications : July 9, 2017
Camera ready submissions due: July 23, 2017
Baseline Weka System for Determining Emotion Intensity:
You are free to build a system from scratch using any available software packages and resources, as long as they are not against the spirit of fair competition. In order to assist testing of ideas, we also provide a baseline emotion intensity system that you can build on. The use of this system is completely optional. The system is available here: https://github.com/felipebravom/AffectiveTweets
Organizers of the shared task:
Saif M. Mohammad
saif.mohammad [at] nrc-cnrc.gc.ca
National Research Council Canada
Felipe Bravo-Marquez
fjb11 [at] students.waikato.ac.nz
The University of Waikato
Alexandra Balahur
alexandra.balahur [at] jrc.ec.europa.eu
European Commission, Brussels
Contact:
Saif M. Mohammad, saif.mohammad [at] nrc-cnrc.gc.ca