We are pleased to announce the opening of the second Dialog State Tracking Challenge (DSTC2), and also the schedule for the third (DSTC3). Complete information, including the challenge handbook, training data, evaluation scripts, and baseline trackers are available on the DSTC2 website:
http://camdial.org/~mh521/dstc/
The Dialog State Tracking Challenge (DSTC) is a research challenge focused on improving the state of the art in tracking the state of spoken dialog systems. State tracking refers to accurately estimating the user's goal as a dialog progresses. Accurate state tracking is desirable because it provides robustness to errors in speech recognition, and helps reduce ambiguity inherent in language within a temporal process like dialog.
In this challenge, participants are given labelled corpora of dialogs to develop state tracking algorithms. The trackers will then be evaluated on a common set of held-out dialogs which are released, un-labelled, during a one week period. This is a corpus-based challenge: participants do not need to implement a speech recognizer, a semantic parser, or an end-to-end dialog system.
The first DSTC recently completed, with 9 teams participating and a total of 27 entries, with 9 papers presented at SIGDIAL 2013, advancing the state-of-the-art in several dimensions. DSTC2 introduces a completely new dataset, in a new domain (restaurant information), with more complicated and dynamic dialog states that may change throughout the dialog.
DSTC2 schedule:
7 October 2013 : Labelled restaurant information train and development set released
20 January 2014 : Unlabelled restaurant information test set released
27 January 2014 : Tracker output on restaurant information test set due
3 February 2014 : Results on restaurant information test set given to participants
5 March 2014 : Approximate SIGdial deadline
Mid-June 2014 : Results presented at SIGDIAL 2014 Conference (to be loosely co-located with ACL)
The training data, scoring scripts, and baselines are available for public download. Prospective participants are strongly encouraged to join the mailing list, to ensure you receive notifications of updates to data or scripts, and are included in discussions about the challenge. To join, email listserv@lists.research.microsoft.com with 'subscribe DSTC' in the body of the message (without quotes).
Feel free to direct questions to the organizers. We hope you will consider participating!
DSTC2/3 organizers
Matt Henderson (lead) - Cambridge University [matthen@gmail.com]
Blaise Thomson - Cambridge University [brmt2@cam.ac.uk]
Jason D. Williams - Microsoft Research [jason.williams@microsoft.com]
DSTC2/3 advisory board
Bill Byrne - University of Cambridge
Paul Crook - Microsoft Research
Maxine Eskenazi - Carnegie Mellon University
Milica Gasic - University of Cambridge
Helen Hastie - Herriot Watt
Kee-Eung Kim - KAIST
Sungjin Lee - Carnegie Mellon University
Oliver Lemon - Herriot Watt
Olivier Pietquin - SUPELEC
Joelle Pineau - McGill University
Deepak Ramachandran - Nuance Communications
Brian Strope - Google
Steve Young - University of Cambridge