论文标题
深度神经网络中训练损失水平集的数值探索
Numerical Exploration of Training Loss Level-Sets in Deep Neural Networks
论文作者
论文摘要
我们提出了一种计算方法，用于凭经验表征深神经网络的训练损失水平集。我们的方法在数值上构建了参数空间中的路径，该路径被限制在固定接近零训练损失的集合中。通过测量该路径内不同点处的正则化功能和测试损失，我们检查了参数空间中不同点的不同点在概括能力方面与相同的固定训练损失进行比较。我们还比较了此方法与更典型的方法一起查找正则点的方法，该方法使用的是目标函数，这些函数是加权训练损失和正则化项的加权总和。我们将尺寸降低应用于经过的路径，以可视化良好的参数空间区域中的损耗级别集。我们的结果提供了有关深神经网络损失格局的新信息，以及减少测试损失的新策略。
We present a computational method for empirically characterizing the training loss level-sets of deep neural networks. Our method numerically constructs a path in parameter space that is constrained to a set with a fixed near-zero training loss. By measuring regularization functions and test loss at different points within this path, we examine how different points in the parameter space with the same fixed training loss compare in terms of generalization ability. We also compare this method for finding regularized points with the more typical method, that uses objective functions which are weighted sums of training loss and regularization terms. We apply dimensionality reduction to the traversed paths in order to visualize the loss level sets in a well-regularized region of parameter space. Our results provide new information about the loss landscape of deep neural networks, as well as a new strategy for reducing test loss.
