Predicting Success in Goal-Driven Human-Human Dialogues

Michael Noseworthy, Jackie Chi Kit Cheung, Joelle Pineau


Abstract
In goal-driven dialogue systems, success is often defined based on a structured definition of the goal. This requires that the dialogue system be constrained to handle a specific class of goals and that there be a mechanism to measure success with respect to that goal. However, in many human-human dialogues the diversity of goals makes it infeasible to define success in such a way. To address this scenario, we consider the task of automatically predicting success in goal-driven human-human dialogues using only the information communicated between participants in the form of text. We build a dataset from stackoverflow.com which consists of exchanges between two users in the technical domain where ground-truth success labels are available. We then propose a turn-based hierarchical neural network model that can be used to predict success without requiring a structured goal definition. We show this model outperforms rule-based heuristics and other baselines as it is able to detect patterns over the course of a dialogue and capture notions such as gratitude.
Anthology ID:
W17-5531
Volume:
Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue
Month:
August
Year:
2017
Address:
Saarbrücken, Germany
Editors:
Kristiina Jokinen, Manfred Stede, David DeVault, Annie Louis
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
253–262
Language:
URL:
https://aclanthology.org/W17-5531
DOI:
10.18653/v1/W17-5531
Bibkey:
Cite (ACL):
Michael Noseworthy, Jackie Chi Kit Cheung, and Joelle Pineau. 2017. Predicting Success in Goal-Driven Human-Human Dialogues. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 253–262, Saarbrücken, Germany. Association for Computational Linguistics.
Cite (Informal):
Predicting Success in Goal-Driven Human-Human Dialogues (Noseworthy et al., SIGDIAL 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-5531.pdf