%0 Conference Proceedings %T A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd %A Miller, Tristan %A Sukhareva, Maria %A Gurevych, Iryna %Y Burstein, Jill %Y Doran, Christy %Y Solorio, Thamar %S Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) %D 2019 %8 June %I Association for Computational Linguistics %C Minneapolis, Minnesota %F miller-etal-2019-streamlined %X The study of argumentation and the development of argument mining tools depends on the availability of annotated data, which is challenging to obtain in sufficient quantity and quality. We present a method that breaks down a popular but relatively complex discourse-level argument annotation scheme into a simpler, iterative procedure that can be applied even by untrained annotators. We apply this method in a crowdsourcing setup and report on the reliability of the annotations obtained. The source code for a tool implementing our annotation method, as well as the sample data we obtained (4909 gold-standard annotations across 982 documents), are freely released to the research community. These are intended to serve the needs of qualitative research into argumentation, as well as of data-driven approaches to argument mining. %R 10.18653/v1/N19-1177 %U https://aclanthology.org/N19-1177 %U https://doi.org/10.18653/v1/N19-1177 %P 1790-1796