论文标题
预先训练的层次变压器无监督的提取性摘要
Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
论文作者
论文摘要
无监督的提取文档摘要旨在从文档中选择重要的句子，而无需在培训期间使用标记的摘要。现有方法主要是基于图形的，句子是节点和边缘权重，该节点和边缘权重以句子相似性衡量。在这项工作中，我们发现变压器的注意力可用于对无监督的提取性摘要进行排名。具体而言，我们首先使用仅使用未标记文档的层次变压器模型预先培训。然后，我们提出了一种使用句子级别的自我训练和训练预训练目标来对句子进行排名的方法。有关CNN/Dailymail和New York Times数据集的实验显示，我们的模型在无监督的摘要方面实现了最先进的性能。我们在实验中还发现，我们的模型较少依赖句子位置。当使用我们的模型的线性组合和最新的无监督模型明确建模句子位置时，我们获得了更好的结果。
Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. In this work, we find that transformer attentions can be used to rank sentences for unsupervised extractive summarization. Specifically, we first pre-train a hierarchical transformer model using unlabeled documents only. Then we propose a method to rank sentences using sentence-level self-attentions and pre-training objectives. Experiments on CNN/DailyMail and New York Times datasets show our model achieves state-of-the-art performance on unsupervised summarization. We also find in experiments that our model is less dependent on sentence positions. When using a linear combination of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results.
