论文标题
稀疏神经网络的非凸正则化
Nonconvex regularization for sparse neural networks
论文作者
论文摘要
凸$ \ ell_1 $使用无限的神经元词典进行正则化，以构建具有所需近似保证的神经网络，但可能会受到任意数量的过度参数化的影响。这可能导致稀疏性丧失，并导致给定数据的网络具有过多的活性神经元，特别是如果数据样本数量较大。作为一种补救措施，在本文中，在浅层relu网络的背景下研究了一种非凸正则化方法：我们证明，与凸方法相反，即使存在无限数据的存在（即，如果知道数据分布，并且考虑了无限样本的限制情况），则任何结果（本地最佳）网络也是有限的）。此外，我们表明，维持有限数据的网络大小上的近似保证和现有界限。
Convex $\ell_1$ regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, we show that approximation guarantees and existing bounds on the network size for finite data are maintained.
