论文标题
变压器是否编码基础本体论？用自然语言探测抽象课程
Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language
论文作者
论文摘要
借助探测的方法学支持（或诊断分类），最近的研究表明，变形金刚在某种程度上编码了句法和语义信息。在这一研究之后，本文旨在将语义探测带到抽象极端，目的是回答以下研究问题：当代变压器的模型能否反映基本的基础本体论？为此，我们提出了一种系统的基础本体论（FO）探测方法，以研究基于变形金刚的模型是否编码抽象的语义信息。遵循不同的预训练和微调制度，我们在三个不同且互补的FO标记实验上对各种大规模语言模型进行了广泛的评估。具体而言，我们介绍并讨论以下结论：（1）探测结果表明，基于变压器的模型偶然编码了培训前专业人士期间与基础本体相关的信息； （2）可以有效地利用此知识的有效构建强大的FO标签器（精度为90％）。
With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an underlying Foundational Ontology? To this end, we present a systematic Foundational Ontology (FO) probing methodology to investigate whether Transformers-based models encode abstract semantic information. Following different pre-training and fine-tuning regimes, we present an extensive evaluation of a diverse set of large-scale language models over three distinct and complementary FO tagging experiments. Specifically, we present and discuss the following conclusions: (1) The probing results indicate that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers (accuracy of 90 percent)can be efficiently built leveraging on this knowledge.
