We are delighted to announce SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Sentences through Narrative Understanding.
Word Sense Disambiguation tasks commonly assume that only one word sense is "correct", but this does not always reflect how humans perceive meaning. Ambiguity, underspecification and subjective factors such as individual linguistic experience can influence which word senses are perceived as plausible in a given context. For these reasons, human intuitions and model predictions may frequently diverge.
This shared task addresses this challenge using the AmbiStory dataset. The dataset consists of five-sentence English short stories, each containing a lexically ambiguous sentence with human plausibility judgements. Surrounding context may provide clues or influence how plausible different word senses appear.
Example:
> "The detectives followed the track."
> Depending on the given context, either the rail track or the piece of evidence sense may be perceived as plausible.
=== Task ===
Given a short story with a lexically ambiguous sentence and a pair of candidate word senses, participating systems must predict the plausibility of each sense on a scale from 1 (implausible) to 5 (highly plausible).
Systems will be evaluated based on two metrics:
* Accuracy within standard deviation (SD): the proportion of predictions that fall within one standard deviation of the average human judgment.
* Ranked correlation: the correlation between model predictions and aggregated human judgments.
=== Participation ===
Website and Data: https://nlu-lab.github.io/semeval.html
Code: https://github.com/Janosch-Gehring/semeval26-05-scripts
Registration and Submission: https://www.codabench.org/competitions/10877/
We invite researchers and students of all experience levels to participate in this task and help us disambiguate the world, story by story!
Do you have any questions? Please join our Google group: https://groups.google.com/g/semeval2026-task5-participants
=== Important Dates ===
Development phase: now -- Jan 10, 2026
Evaluation phase: Jan 10 -- Jan 31, 2026
Feb 2026 (tentative): Shared task papers due
Mar 2026 (tentative): Notification of acceptance
Apr 2026 (tentative): Camera-ready submission
Summer 2026: SemEval workshop
=== Organizing Committee ===
Janosch Gehring, University of Technology Nuremberg
Selina Meyer, University of Technology Nuremberg
Michael Roth, University of Technology Nuremberg