论文标题
正交梯度加快神经网络优化
Orthogonalising gradients to speed up neural network optimisation
论文作者
论文摘要
可以通过在优化步骤之前对梯度进行正交来加强神经网络的优化，从而确保学到的表示形式的多样化。我们正交将层组件/过滤器相对于彼此的梯度分开以分离中间表示形式。我们的正交化方法可以更灵活地使用权重，而将权重限制为正交子空间。我们在ImageNet和CIFAR-10上测试了这种方法，导致学习时间大大减少，并在半监督的学习Barlowtwins上获得了加速。我们获得了与SGD相似的精度，而无需微调，并且对于天真选择的超参数而言，我们获得了更好的准确性。
The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We orthogonalise the gradients of the layer's components/filters with respect to each other to separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for naïvely chosen hyper-parameters.
