An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining

Yifan Peng, Qingyu Chen, Zhiyong Lu


Abstract
Multi-task learning (MTL) has achieved remarkable success in natural language processing applications. In this work, we study a multi-task learning model with multiple decoders on varieties of biomedical and clinical natural language processing tasks such as text similarity, relation extraction, named entity recognition, and text inference. Our empirical results demonstrate that the MTL fine-tuned models outperform state-of-the-art transformer models (e.g., BERT and its variants) by 2.0% and 1.3% in biomedical and clinical domain adaptation, respectively. Pairwise MTL further demonstrates more details about which tasks can improve or decrease others. This is particularly helpful in the context that researchers are in the hassle of choosing a suitable model for new problems. The code and models are publicly available at https://github.com/ncbi-nlp/bluebert.
Anthology ID:
2020.bionlp-1.22
Volume:
Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing
Month:
July
Year:
2020
Address:
Online
Editors:
Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, Junichi Tsujii
Venue:
BioNLP
SIG:
SIGBIOMED
Publisher:
Association for Computational Linguistics
Note:
Pages:
205–214
Language:
URL:
https://aclanthology.org/2020.bionlp-1.22
DOI:
10.18653/v1/2020.bionlp-1.22
Bibkey:
Cite (ACL):
Yifan Peng, Qingyu Chen, and Zhiyong Lu. 2020. An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 205–214, Online. Association for Computational Linguistics.
Cite (Informal):
An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining (Peng et al., BioNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.bionlp-1.22.pdf
Code
 ncbi-nlp/bluebert
Data
BC5CDRBLUE