论文标题
知识的调查增强了预训练的语言模型
A Survey of Knowledge Enhanced Pre-trained Language Models
论文作者
论文摘要
通过自我监督学习方法在大型文本语料库中接受培训的预训练的语言模型（PLM），在自然语言处理（NLP）的各种任务方面取得了有希望的表现。但是，尽管具有巨大参数的PLM可以有效地拥有从大规模培训文本中学到的丰富知识，并在微调阶段受益于下游任务，但由于缺乏外部知识，它们仍然存在一些局限性，例如推理能力差。研究一直致力于将知识纳入PLM，以解决这些问题。在本文中，我们对知识增强的预训练的语言模型（KE-PLM）进行了全面审查，以清楚地了解这一蓬勃发展的领域。我们分别引入适当的分类法，以了解自然语言理解（NLU）和自然语言生成（NLG），以突出NLP的这两个主要任务。对于NLU，我们将知识的类型分为四类：语言知识，文本知识，知识图（kg）和规则知识。 NLG的KE-PLM分为基于KG的基于KG和基于检索的方法。最后，我们指出了KE-PLM的一些有希望的未来方向。
Pre-trained Language Models (PLMs) which are trained on large text corpus via self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Research has been dedicated to incorporating knowledge into PLMs to tackle these issues. In this paper, we present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight these two main tasks of NLP. For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.
