Rapid advances in generative models for both language and vision have made these models increasingly popular in both the public and private sectors. For example, governments use generative models such as chatbots to better serve citizens. As such, it is critical that we not only evaluate whether these models are safe enough to deploy, but also ensure that the evaluation systems themselves are reliable. Oftentimes, humans are used to evaluate these models. Our goal is to call attention to the discussion on how to best perform reliable human evaluations of generative models. Through this discussion, we aim to uplift cutting edge research and engage stakeholders in dialogue on how to address these challenges from their perspective. Critical considerations of safe deployment include reproducibility and trustworthiness of an evaluation, assessment of human-AI interaction when predictions lead to policy decisions, and value-alignment of these systems.
In partnership with the Day One Project --- a project of the Federation of American Scientists, an impact-driven policy think tank that helps subject matter experts become policy entrepreneurs--- we will select a few papers with clear policy implications and recommendations, invite authors to write policy memos, and work to implement those policy recommendations. Finally, we will capture the discussions that happen during our panels in a paper that will summarize the workshop recommendations and seek to publish that work for scholarly record.
Topics of interest include but are not limited to the following:
1. Experimental design and methods for human evaluations
2. Role of human evaluation in the context of value alignment of large
generative models
3. Designing testbeds for evaluating generative models
4. Reproducibility of human evaluations
5. Ethical considerations in human evaluation of computational systems
6. Quality assurance for human evaluation
7. Issues in meta-evaluation of automatic metrics by correlation with human
evaluations
8. Methods for assessing the quality and the reliability of human evaluations
Organizers:
Divyansh Kaushik (Carnegie Mellon University)
Jennifer Hsia (Carnegie Mellon University)
Jessica Huynh (Carnegie Mellon University)
Yonadav Shavit (Harvard University)
Samuel R. Bowman (New York University)
Ting-Hao (Kenneth) Huang (Penn State University)
Douwe Kiela (Hugging Face)
Zachary Lipton (Carnegie Mellon University)
Eric Smith (Facebook AI Research)
Important dates
New Submission deadline: September 22, 2022
Acceptance notifications: October 20, 2022
Camera-ready deadline: November 3, 2022
Workshop date: December 3, 2022
Submission: https://openreview.net/group?id=NeurIPS.cc/2022/Workshop/HEGM