Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.

Tal August, Maarten Sap, Elizabeth Clark, Katharina Reinecke, Noah A. Smith


Abstract
Current story writing or story editing systems rely on human judgments of story quality for evaluating performance, often ignoring the subjectivity in ratings. We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task. To study this effect, we create and release STORIESINTHEWILD, containing 1,630 stories collected on a volunteer-based crowdsourcing platform. Each story is rated by three different readers, and comes paired with the author’s and reader’s age, gender, and personality. Our findings show significant effects of authors’ and readers’ identities, as well as writing setup, on story writing and ratings. Notably, compared to younger readers, readers age 45 and older consider stories significantly less creative and less entertaining. Readers also prefer stories written all at once, rather than in chunks, finding them more coherent and creative. We also observe linguistic differences associated with authors’ demographics (e.g., older authors wrote more vivid and emotional stories). Our findings suggest that reader and writer demographics, as well as writing setup, should be accounted for in story writing evaluations.
Anthology ID:
2020.nuse-1.6
Volume:
Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events
Month:
July
Year:
2020
Address:
Online
Editors:
Claire Bonial, Tommaso Caselli, Snigdha Chaturvedi, Elizabeth Clark, Ruihong Huang, Mohit Iyyer, Alejandro Jaimes, Heng Ji, Lara J. Martin, Ben Miller, Teruko Mitamura, Nanyun Peng, Joel Tetreault
Venues:
NUSE | WNU
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
46–54
Language:
URL:
https://aclanthology.org/2020.nuse-1.6
DOI:
10.18653/v1/2020.nuse-1.6
Bibkey:
Cite (ACL):
Tal August, Maarten Sap, Elizabeth Clark, Katharina Reinecke, and Noah A. Smith. 2020. Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 46–54, Online. Association for Computational Linguistics.
Cite (Informal):
Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus. (August et al., NUSE-WNU 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.nuse-1.6.pdf
Software:
 2020.nuse-1.6.Software.zip
Dataset:
 2020.nuse-1.6.Dataset.pdf
Video:
 http://slideslive.com/38929745