Simon Keizer


2023

pdf bib
Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
Norbert Braunschweiler | Rama Doddipatla | Simon Keizer | Svetlana Stoyanchev
Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!

In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus of task-oriented dialogues in four social service domains previously used in the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded in multiple documents providing relevant information. We generate dialogue completion responses by prompting a ChatGPT model, using two methods: Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT model pre-training while LlamaIndex also extracts relevant information from documents. Observing that document-grounded response generation via LLMs cannot be adequately assessed by automatic evaluation metrics as they are significantly more verbose, we perform a human evaluation where annotators rate the output of the shared task winning system, the two ChatGPT variants outputs, and human responses. While both ChatGPT variants are more likely to include information not present in the relevant segments, possibly including a presence of hallucinations, they are rated higher than both the shared task winning system and human responses.

2022

pdf bib
Combining Structured and Unstructured Knowledge in an Interactive Search Dialogue System
Svetlana Stoyanchev | Suraj Pandey | Simon Keizer | Norbert Braunschweiler | Rama Sanand Doddipatla
Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue

Users of interactive search dialogue systems specify their preferences with natural language utterances. However, a schema-driven system is limited to handling the preferences that correspond to the predefined database content. In this work, we present a methodology for extending a schema-driven interactive search dialogue system with the ability to handle unconstrained user preferences. Using unsupervised semantic similarity metrics and the text snippets associated with the search items, the system identifies suitable items for the user’s unconstrained natural language query. In crowd-sourced evaluation, the users chat with our extended restaurant search system. Based on objective metrics and subjective user ratings, we demonstrate the feasibility of using an unsupervised low latency approach to extend a schema-driven search dialogue system to handle unconstrained user preferences.

2020

pdf bib
The ISO Standard for Dialogue Act Annotation, Second Edition
Harry Bunt | Volha Petukhova | Emer Gilmartin | Catherine Pelachaud | Alex Fang | Simon Keizer | Laurent Prévot
Proceedings of the Twelfth Language Resources and Evaluation Conference

ISO standard 24617-2 for dialogue act annotation, established in 2012, has in the past few years been used both in corpus annotation and in the design of components for spoken and multimodal dialogue systems. This has brought some inaccuracies and undesirbale limitations of the standard to light, which are addressed in a proposed second edition. This second edition allows a more accurate annotation of dependence relations and rhetorical relations in dialogue. Following the ISO 24617-4 principles of semantic annotation, and borrowing ideas from EmotionML, a triple-layered plug-in mechanism is introduced which allows dialogue act descriptions to be enriched with information about their semantic content, about accompanying emotions, and other information, and allows the annotation scheme to be customised by adding application-specific dialogue act types.

2019

pdf bib
User Evaluation of a Multi-dimensional Statistical Dialogue System
Simon Keizer | Ondřej Dušek | Xingkun Liu | Verena Rieser
Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue

We present the first complete spoken dialogue system driven by a multiimensional statistical dialogue manager. This framework has been shown to substantially reduce data needs by leveraging domain-independent dimensions, such as social obligations or feedback, which (as we show) can be transferred between domains. In this paper, we conduct a user study and show that the performance of a multi-dimensional system, which can be adapted from a source domain, is equivalent to that of a one-dimensional baseline, which can only be trained from scratch.

2018

pdf bib
Downward Compatible Revision of Dialogue Annotation
Harry Bunt | Emer Gilmartin | Simon Keizer | Catherine Pelachaud | Volha Petukhova | Laurent Prévot | Mariët Theune
Proceedings of the 14th Joint ACL-ISO Workshop on Interoperable Semantic Annotation

2017

pdf bib
Evaluating Persuasion Strategies and Deep Reinforcement Learning methods for Negotiation Dialogue agents
Simon Keizer | Markus Guhe | Heriberto Cuayáhuitl | Ioannis Efstathiou | Klaus-Peter Engelbrecht | Mihai Dobre | Alex Lascarides | Oliver Lemon
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers

In this paper we present a comparative evaluation of various negotiation strategies within an online version of the game “Settlers of Catan”. The comparison is based on human subjects playing games against artificial game-playing agents (‘bots’) which implement different negotiation dialogue strategies, using a chat dialogue interface to negotiate trades. Our results suggest that a negotiation strategy that uses persuasion, as well as a strategy that is trained from data using Deep Reinforcement Learning, both lead to an improved win rate against humans, compared to previous rule-based and supervised learning baseline dialogue negotiators.

2013

pdf bib
Training and evaluation of an MDP model for social multi-user human-robot interaction
Simon Keizer | Mary Ellen Foster | Oliver Lemon | Andre Gaschler | Manuel Giuliani
Proceedings of the SIGDIAL 2013 Conference

2011

pdf bib
Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results
Alan W Black | Susanne Burger | Alistair Conkie | Helen Hastie | Simon Keizer | Oliver Lemon | Nicolas Merigaud | Gabriel Parent | Gabriel Schubiner | Blaise Thomson | Jason D. Williams | Kai Yu | Steve Young | Maxine Eskenazi
Proceedings of the SIGDIAL 2011 Conference

pdf bib
Adaptive Information Presentation for Spoken Dialogue Systems: Evaluation with real users
Verena Rieser | Simon Keizer | Oliver Lemon | Xingkun Liu
Proceedings of the 13th European Workshop on Natural Language Generation

2010

pdf bib
Parameter estimation for agenda-based user simulation
Simon Keizer | Milica Gašić | Filip Jurčíček | François Mairesse | Blaise Thomson | Kai Yu | Steve Young
Proceedings of the SIGDIAL 2010 Conference

pdf bib
Gaussian Processes for Fast Policy Optimisation of POMDP-based Dialogue Managers
Milica Gašić | Filip Jurčíček | Simon Keizer | Francois Mairesse | Blaise Thomson | Kai Yu | Steve Young
Proceedings of the SIGDIAL 2010 Conference

pdf bib
Phrase-Based Statistical Language Generation Using Graphical Models and Active Learning
François Mairesse | Milica Gašić | Filip Jurčíček | Simon Keizer | Blaise Thomson | Kai Yu | Steve Young
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics

2009

pdf bib
k-Nearest Neighbor Monte-Carlo Control Algorithm for POMDP-Based Dialogue Systems
Fabrice Lefèvre | Milica Gašić | Filip Jurčíček | Simon Keizer | François Mairesse | Blaise Thomson | Kai Yu | Steve Young
Proceedings of the SIGDIAL 2009 Conference

2008

pdf bib
Training and Evaluation of the HIS POMDP Dialogue System in Noise
Milica Gašić | Simon Keizer | Francois Mairesse | Jost Schatzmann | Blaise Thomson | Kai Yu | Steve Young
Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue

2007

pdf bib
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue
Harry Bunt | Simon Keizer | Tim Paek
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
Evaluating Combinations of Dialogue Acts for Generation
Simon Keizer | Harry Bunt
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
An Empirically Based Computational Model of Grounding in Dialogue
Harry Bunt | Roser Morante | Simon Keizer
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
Dialogue Simulation and Context Dynamics for Dialogue Management
Simon Keizer | Roser Morante
Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA 2007)

2006

pdf bib
Multidimensional Dialogue Management
Simon Keizer | Harry Bunt
Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue

2002

pdf bib
Dialogue Act Recognition with Bayesian Networks for Dutch Dialogues
Simon Keizer | Rieks op den Akker | Anton Nijholt
Proceedings of the Third SIGdial Workshop on Discourse and Dialogue