Effective Convolutional Attention Network for Multi-label Clinical Document Classification

Yang Liu, Hua Cheng, Russell Klopfer, Matthew R. Gormley, Thomas Schaaf


Abstract
Multi-label document classification (MLDC) problems can be challenging, especially for long documents with a large label set and a long-tail distribution over labels. In this paper, we present an effective convolutional attention network for the MLDC problem with a focus on medical code prediction from clinical documents. Our innovations are three-fold: (1) we utilize a deep convolution-based encoder with the squeeze-and-excitation networks and residual networks to aggregate the information across the document and learn meaningful document representations that cover different ranges of texts; (2) we explore multi-layer and sum-pooling attention to extract the most informative features from these multi-scale representations; (3) we combine binary cross entropy loss and focal loss to improve performance for rare labels. We focus our evaluation study on MIMIC-III, a widely used dataset in the medical domain. Our models outperform prior work on medical coding and achieve new state-of-the-art results on multiple metrics. We also demonstrate the language independent nature of our approach by applying it to two non-English datasets. Our model outperforms prior best model and a multilingual Transformer model by a substantial margin.
Anthology ID:
2021.emnlp-main.481
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5941–5953
Language:
URL:
https://aclanthology.org/2021.emnlp-main.481
DOI:
10.18653/v1/2021.emnlp-main.481
Bibkey:
Cite (ACL):
Yang Liu, Hua Cheng, Russell Klopfer, Matthew R. Gormley, and Thomas Schaaf. 2021. Effective Convolutional Attention Network for Multi-label Clinical Document Classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5941–5953, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Effective Convolutional Attention Network for Multi-label Clinical Document Classification (Liu et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.481.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.481.mp4
Data
MIMIC-III