A pre-training and self-training approach for biomedical named entity recognition

Gao, Shang; Kotevska, Olivera; Sorokine, Alexandre; Christian, J. Blair; Fiorini, ed., Nicolas

doi:10.1371/journal.pone.0246310

Title: A pre-training and self-training approach for biomedical named entity recognition

Journal Article · Tue Feb 09 00:00:00 EST 2021 · PLoS ONE

DOI:https://doi.org/10.1371/journal.pone.0246310· OSTI ID:1765039

; Kotevska, Olivera;

;

; Fiorini, ed., Nicolas

Named entity recognition (NER) is a key component of many scientific literature mining tasks, such as information retrieval, information extraction, and question answering; however, many modern approaches require large amounts of labeled training data in order to be effective. This severely limits the effectiveness of NER models in applications where expert annotations are difficult and expensive to obtain. In this work, we explore the effectiveness of transfer learning and semi-supervised self-training to improve the performance of NER models in biomedical settings with very limited labeled data (250-2000 labeled samples). We first pre-train a BiLSTM-CRF and a BERT model on a very large general biomedical NER corpus such as MedMentions or Semantic Medline, and then we fine-tune the model on a more specific target NER task that has very limited training data; finally, we apply semi-supervised self-training using unlabeled data to further boost model performance. We show that in NER tasks that focus on common biomedical entity types such as those in the Unified Medical Language System (UMLS), combining transfer learning with self-training enables a NER model such as a BiLSTM-CRF or BERT to obtain similar performance with the same model trained on 3x-8x the amount of labeled data. We further show that our approach can also boost performance in a low-resource application where entities types are more rare and not specifically covered in UMLS.

View Journal Article

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1765039

Alternate ID(s):: OSTI ID: 1766402

Journal Information:: PLoS ONE, Journal Name: PLoS ONE Vol. 16 Journal Issue: 2; ISSN 1932-6203

Publisher:: Public Library of ScienceCopyright Statement

Country of Publication:: United States

Language:: English

References (44)

An overview of MetaMap: historical perspective and recent advances Aronson, Alan R.; Lang, François-Michel Journal of the American Medical Informatics Association, Vol. 17, Issue 3 https://doi.org/10.1136/jamia.2009.002733	journal	May 2010
Rule-based pattern extractor and named entity recognition: A hybrid approach Sari, Yunita; Hassan, Mohd Fadzil; Zamin, Norshuhani 2010 International Symposium on Information Technology (ITSim 2010) https://doi.org/10.1109/ITSIM.2010.5561392	conference	June 2010
SemMedDB: a PubMed-scale repository of biomedical semantic predications Kilicoglu, H.; Shin, D.; Fiszman, M. Bioinformatics, Vol. 28, Issue 23 https://doi.org/10.1093/bioinformatics/bts591	journal	October 2012
Towards reliable named entity recognition in the biomedical domain Giorgi, John M.; Bader, Gary D. Bioinformatics, Vol. 36, Issue 1 https://doi.org/10.1093/bioinformatics/btz504	journal	June 2019
Transfer Learning in Biomedical Named Entity Recognition: An Evaluation of BERT in the PharmaCoNER task Sun, Cong; Yang, Zhihao Proceedings of The 5th Workshop on BioNLP Open Shared Tasks https://doi.org/10.18653/v1/D19-5715	conference	January 2019
A context pattern induction method for named entity extraction Talukdar, Partha Pratim; Brants, Thorsten; Liberman, Mark Proceedings of the Tenth Conference on Computational Natural Language Learning - CoNLL-X '06 https://doi.org/10.3115/1596276.1596303	conference	January 2006
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF Ma, Xuezhe; Hovy, Eduard Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) https://doi.org/10.18653/v1/P16-1101	conference	January 2016
Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets Peng, Yifan; Yan, Shankai; Lu, Zhiyong Proceedings of the 18th BioNLP Workshop and Shared Task https://doi.org/10.18653/v1/W19-5006	conference	January 2019
A Bidirectional LSTM and Conditional Random Fields Approach to Medical Named Entity Recognition Xu, Kai; Zhou, Zhanfan; Hao, Tianyong Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2017 https://doi.org/10.1007/978-3-319-64861-3_33	book	August 2017
HUNER: improving biomedical NER with pretraining Weber, Leon; Münchmeyer, Jannes; Rocktäschel, Tim Bioinformatics, Vol. 36, Issue 1 https://doi.org/10.1093/bioinformatics/btz528	journal	June 2019
A Neural Named Entity Recognition and Multi-Type Normalization Tool for Biomedical Text Mining Kim, Donghyeon; Lee, Jinhyuk; So, Chan Ho IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2920708	journal	January 2019
Semantic MEDLINE: An advanced information management application for biomedicine Rindflesch, Thomas C.; Kilicoglu, Halil; Fiszman, Marcelo Information Services & Use, Vol. 31, Issue 1-2 https://doi.org/10.3233/ISU-2011-0627	journal	September 2011
Multi-domain evaluation framework for named entity recognition tools Abdallah, Zahraa S.; Carman, Mark; Haffari, Gholamreza Computer Speech & Language, Vol. 43 https://doi.org/10.1016/j.csl.2016.10.003	journal	May 2017
MetaNER: Named Entity Recognition with Meta-Learning Li, Jing; Shang, Shuo; Shao, Ling WWW '20: The Web Conference 2020, Proceedings of The Web Conference 2020 https://doi.org/10.1145/3366423.3380127	conference	April 2020
Rule-Based Method for Entity Resolution Li, Lingli; Li, Jianzhong; Gao, Hong IEEE Transactions on Knowledge and Data Engineering, Vol. 27, Issue 1 https://doi.org/10.1109/TKDE.2014.2320713	journal	January 2015
Few-Shot Named Entity Recognition via Meta-Learning Li, Jing; Chiu, Billy; Feng, Shanshan IEEE Transactions on Knowledge and Data Engineering https://doi.org/10.1109/TKDE.2020.3038670	journal	January 2020
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift Ruder, Sebastian; Plank, Barbara Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) https://doi.org/10.18653/v1/P18-1096	conference	January 2018
BioBERT: a pre-trained biomedical language representation model for biomedical text mining Lee, Jinhyuk; Yoon, Wonjin; Kim, Sungdong Bioinformatics https://doi.org/10.1093/bioinformatics/btz682	journal	September 2019
Deep Model Based Transfer and Multi-Task Learning for Biological Image Analysis Zhang, Wenlu; Li, Rongjian; Zeng, Tao IEEE Transactions on Big Data, Vol. 6, Issue 2 https://doi.org/10.1109/TBDATA.2016.2573280	journal	June 2020
Biomedical Named Entity Recognition with Multilingual BERT Hakala, Kai; Pyysalo, Sampo Proceedings of The 5th Workshop on BioNLP Open Shared Tasks https://doi.org/10.18653/v1/D19-5709	conference	January 2019
DTranNER: biomedical named entity recognition with deep learning-based label-label transition model Hong, S. K.; Lee, Jae-Gil BMC Bioinformatics, Vol. 21, Issue 1 https://doi.org/10.1186/s12859-020-3393-1	journal	February 2020
BioWordVec, improving biomedical word embeddings with subword information and MeSH Zhang, Yijia; Chen, Qingyu; Yang, Zhihao Scientific Data, Vol. 6, Issue 1 https://doi.org/10.1038/s41597-019-0055-0	journal	May 2019
ChemSpot: a hybrid system for chemical named entity recognition Rocktäschel, Tim; Weidlich, Michael; Leser, Ulf Bioinformatics, Vol. 28, Issue 12 https://doi.org/10.1093/bioinformatics/bts183	journal	April 2012
Artificial intelligence to organize patient portal messages: a journey from an ensemble deep learning text classification to rule-based named entity recognition Tafti, Ahmad P.; Fu, Sunyang; Khurana, Aditya 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) https://doi.org/10.1109/BIBM47256.2019.8982942	conference	November 2019
Neural Architectures for Named Entity Recognition Lample, Guillaume; Ballesteros, Miguel; Subramanian, Sandeep Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies https://doi.org/10.18653/v1/N16-1030	conference	January 2016
A Survey on Transfer Learning Pan, Sinno Jialin; Yang, Qiang IEEE Transactions on Knowledge and Data Engineering, Vol. 22, Issue 10 https://doi.org/10.1109/TKDE.2009.191	journal	October 2010
Transfer learning for biomedical named entity recognition with neural networks Giorgi, John M.; Bader, Gary D. Bioinformatics, Vol. 34, Issue 23 https://doi.org/10.1093/bioinformatics/bty449	journal	June 2018
Better Modeling of Incomplete Annotations for Named Entity Recognition Jie, Zhanming; Xie, Pengjun; Lu, Wei Proceedings of the 2019 Conference of the North https://doi.org/10.18653/v1/N19-1079	conference	January 2019
Extraction of protein interaction information from unstructured text using a context-free grammar Temkin, J. M.; Gilder, M. R. Bioinformatics, Vol. 19, Issue 16 https://doi.org/10.1093/bioinformatics/btg279	journal	October 2003
Transfer Learning for Low-Resource Neural Machine Translation Zoph, Barret; Yuret, Deniz; May, Jonathan Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/D16-1163	conference	January 2016
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing Neumann, Mark; King, Daniel; Beltagy, Iz Proceedings of the 18th BioNLP Workshop and Shared Task https://doi.org/10.18653/v1/W19-5034	conference	January 2019
Scientific Information Extraction with Semi-supervised Neural Tagging Luan, Yi; Ostendorf, Mari; Hajishirzi, Hannaneh Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/D17-1279	conference	January 2017
A rule-based named-entity recognition method for knowledge extraction of evidence-based dietary recommendations Eftimov, Tome; Koroušić Seljak, Barbara; Korošec, Peter PLOS ONE, Vol. 12, Issue 6 https://doi.org/10.1371/journal.pone.0179488	journal	June 2017
Using machine learning to maintain rule-based named-entity recognition and classification systems Petasis, Georgios; Vichot, Frantz; Wolinski, Francis Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01 https://doi.org/10.3115/1073012.1073067	conference	January 2001
Dynamic Transfer Learning for Named Entity Recognition Bhatia, Parminder; Arumae, Kristjan; Busra Celikkaya, E. Precision Health and Medicine: A Digital Revolution in Healthcare https://doi.org/10.1007/978-3-030-24409-5_7	book	August 2019
Effect of Character and Word Features in Bidirectional LSTM-CRF for NER Ronran, Chirawan; Lee, Seungwoo 2020 IEEE International Conference on Big Data and Smart Computing (BigComp) https://doi.org/10.1109/BigComp48618.2020.00132	conference	February 2020
A survey on semi-supervised learning van Engelen, Jesper E.; Hoos, Holger H. Machine Learning, Vol. 109, Issue 2 https://doi.org/10.1007/s10994-019-05855-6	journal	November 2019
A novel method for prokaryotic promoter prediction based on DNA stability Kanhere, Aditi; Bansal, Manju BMC Bioinformatics, Vol. 6, Issue 1, 1 https://doi.org/10.1186/1471-2105-6-1	journal	January 2005
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning Miyato, Takeru; Maeda, Shin-Ichi; Koyama, Masanori IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, Issue 8 https://doi.org/10.1109/TPAMI.2018.2858821	journal	August 2019
Transformers: State-of-the-Art Natural Language Processing Wolf, Thomas; Debut, Lysandre; Sanh, Victor Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations https://doi.org/10.18653/v1/2020.emnlp-demos.6	conference	January 2020
A Bootstrapping Approach With CRF and Deep Learning Models for Improving the Biomedical Named Entity Recognition in Multi-Domains Kim, Juae; Ko, Youngjoong; Seo, Jungyun IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2914168	journal	January 2019
Pre-trained models for natural language processing: A survey Qiu, XiPeng; Sun, TianXiang; Xu, YiGe Science China Technological Sciences, Vol. 63, Issue 10 https://doi.org/10.1007/s11431-020-1647-3	journal	September 2020
A simple semi-supervised algorithm for named entity recognition Liao, Wenhui; Veeramachaneni, Sriharsha Proceedings of the NAACL HLT 2009 Workshop on Semi-Supervised Learning for Natural Language Processing - SemiSupLearn '09 https://doi.org/10.3115/1621829.1621837	conference	January 2009
Semi-supervised learning for named entity recognition using weakly labeled training data Zafarian, Atefeh; Rokni, Ali; Khadivi, Shahram 2015 International Symposium on Artificial Intelligence and Signal Processing (AISP), 2015 The International Symposium on Artificial Intelligence and Signal Processing (AISP) https://doi.org/10.1109/AISP.2015.7123504	conference	March 2015

Similar Records

Creating Training Data for Scientific Named Entity Recognition with Minimal Human Effort

Conference · Tue Jan 01 00:00:00 EST 2019 · OSTI ID:1765039

Tchoua, Roselyne B.; Ajith, Aswathy; Hong, Zhi; +7 more

CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model

Journal Article · Thu Nov 04 00:00:00 EDT 2021 · Journal of Cybersecurity and Privacy · OSTI ID:1765039

Ameri, Kimia; Hempel, Michael; Sharif, Hamid; +2 more

Unified Medical Language System resources improve sieve-based generation and Bidirectional Encoder Representations from Transformers (BERT)–based ranking for concept normalization

Journal Article · Mon Jul 27 00:00:00 EDT 2020 · Journal of the American Medical Informatics Association · OSTI ID:1765039

Xu, Dongfang; Gopale, Manoj; Zhang, Jiacheng; +3 more

Related Subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Title: A pre-training and self-training approach for biomedical named entity recognition

Citation Formats

References (44)

Similar Records

Related Subjects