Evaluating Improvised Hip Hop Lyrics - Challenges and Observations

Karteek Addanki, Dekai Wu


Abstract
We investigate novel challenges involved in comparing model performance on the task of improvising responses to hip hop lyrics and discuss observations regarding inter-evaluator agreement on judging improvisation quality. We believe the analysis serves as a first step toward designing robust evaluation strategies for improvisation tasks, a relatively neglected area to date. Unlike most natural language processing tasks, improvisation tasks suffer from a high degree of subjectivity, making it difficult to design discriminative evaluation strategies to drive model development. We propose a simple strategy with fluency and rhyming as the criteria for evaluating the quality of generated responses, which we apply to both our inversion transduction grammar based FREESTYLE hip hop challenge-response improvisation system, as well as various contrastive systems. We report inter-evaluator agreement for both English and French hip hop lyrics, and analyze correlation with challenge length. We also compare the extent of agreement in evaluating fluency with that of rhyming, and quantify the difference in agreement with and without precise definitions of evaluation criteria.
Anthology ID:
L14-1097
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
4616–4623
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1139_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Karteek Addanki and Dekai Wu. 2014. Evaluating Improvised Hip Hop Lyrics - Challenges and Observations. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4616–4623, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
Evaluating Improvised Hip Hop Lyrics - Challenges and Observations (Addanki & Wu, LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1139_Paper.pdf