论文标题
双线性优化的同性下降
Cogradient Descent for Bilinear Optimization
论文作者
论文摘要
传统的学习方法通过独立的两个本质上耦合因子来简化双线性模型，从而降低了优化过程。原因之一是由于异步梯度下降而导致的训练不足，这导致耦合变量的梯度消失。在本文中，我们基于一个理论框架，引入了一种可行的下降算法（COGD）来解决双线性问题，以通过投影函数来协调隐藏变量的梯度。我们通过考虑其与另一个变量的耦合关系来解决一个变量，从而导致同步梯度下降以促进优化过程。我们的算法应用于在稀疏性约束下解决一个变量的问题，该变量被广泛用于学习范式中。我们考虑了一系列广泛的应用程序，包括图像重建，介入和网络修剪，我们验证了我们的COGD。实验表明，它可以通过显着的余量提高最新技术。
Conventional learning methods simplify the bilinear model by regarding two intrinsically coupled factors independently, which degrades the optimization procedure. One reason lies in the insufficient training due to the asynchronous gradient descent, which results in vanishing gradients for the coupled variables. In this paper, we introduce a Cogradient Descent algorithm (CoGD) to address the bilinear problem, based on a theoretical framework to coordinate the gradient of hidden variables via a projection function. We solve one variable by considering its coupling relationship with the other, leading to a synchronous gradient descent to facilitate the optimization procedure. Our algorithm is applied to solve problems with one variable under the sparsity constraint, which is widely used in the learning paradigm. We validate our CoGD considering an extensive set of applications including image reconstruction, inpainting, and network pruning. Experiments show that it improves the state-of-the-art by a significant margin.
