论文标题
初始化时稳健的修剪
Robust Pruning at Initialization
论文作者
论文摘要
过度参数化神经网络（NN）显示最先进的性能。但是，越来越需要在具有有限计算资源的设备上使用机器学习应用程序，需要较小，节能，神经网络。一种流行的方法包括使用修剪技术。尽管这些技术传统上一直集中在修剪预培训的NN上（Lecun等，1990; Hassibi等，1993），但Lee等人的最新工作。 （2018）在初始化时进行修剪时显示出令人鼓舞的结果。但是，对于深度NN，由于所得的修剪网络可能难以训练，例如，这些程序仍然不令人满意，例如，它们不会阻止一层完全修剪。在本文中，我们对稀疏体系结构的初始化和培训进行了全面的理论分析，对幅度和基于梯度的修剪。这使我们能够提出新颖的原则方法，这些方法我们在各种NN体系结构上对这些方法进行了验证。
Overparameterized Neural Networks (NN) display state-of-the-art performance. However, there is a growing need for smaller, energy-efficient, neural networks tobe able to use machine learning applications on devices with limited computational resources. A popular approach consists of using pruning techniques. While these techniques have traditionally focused on pruning pre-trained NN (LeCun et al.,1990; Hassibi et al., 1993), recent work by Lee et al. (2018) has shown promising results when pruning at initialization. However, for Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned. In this paper, we provide a comprehensive theoretical analysis of Magnitude and Gradient based pruning at initialization and training of sparse architectures. This allows us to propose novel principled approaches which we validate experimentally on a variety of NN architectures.
