Automated coherence measures fail to index thought disorder in individuals at risk for psychosis

Kasia Hitczenko, Henry Cowan, Vijay Mittal, Matthew Goldrick


Abstract
Thought disorder – linguistic disturbances including incoherence and derailment of topic – is seen in individuals both with and at risk for psychosis. Methods from computational linguistics have increasingly sought to quantify thought disorder to detect group differences between clinical populations and healthy controls. While previous work has been quite successful at these classification tasks, the lack of interpretability of the computational metrics has made it unclear whether they are in fact measuring thought disorder. In this paper, we dive into these measures to try to better understand what they reflect. While we find group differences between at-risk and healthy control populations, we also find that the measures mostly do not correlate with existing measures of thought disorder symptoms (what they are intended to measure), but rather correlate with surface properties of the speech (e.g., sentence length) and sociodemographic properties of the speaker (e.g., race). These results highlight the importance of considering interpretability and front and center as the field continues to grow. Ethical use of computational measures like those studied here – especially in the high-stakes context of clinical care – requires us to devote substantial attention to potential biases in our measures.
Anthology ID:
2021.clpsych-1.16
Volume:
Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access
Month:
June
Year:
2021
Address:
Online
Editors:
Nazli Goharian, Philip Resnik, Andrew Yates, Molly Ireland, Kate Niederhoffer, Rebecca Resnik
Venue:
CLPsych
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
129–150
Language:
URL:
https://aclanthology.org/2021.clpsych-1.16
DOI:
10.18653/v1/2021.clpsych-1.16
Bibkey:
Cite (ACL):
Kasia Hitczenko, Henry Cowan, Vijay Mittal, and Matthew Goldrick. 2021. Automated coherence measures fail to index thought disorder in individuals at risk for psychosis. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 129–150, Online. Association for Computational Linguistics.
Cite (Informal):
Automated coherence measures fail to index thought disorder in individuals at risk for psychosis (Hitczenko et al., CLPsych 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.clpsych-1.16.pdf