论文标题
对自然梯度的几何理解
A Geometric Understanding of Natural Gradient
论文作者
论文摘要
尽管从理论和经验的角度都对自然梯度进行了广泛的研究，但我们认为，关于无限尺寸函数空间中梯度存在的一些基本理论问题仍然尚未被逐渐解散。我们通过提供几何观点和数学框架来研究这些问题，以研究比现有研究更完整和严格的自然梯度。我们的结果还建立了自然梯度与RKHS理论之间的新联系，特别是与神经切线内核（NTK）之间的联系。根据我们的理论框架，我们得出了由Sobolev指标引起的新的自然梯度系列，并开发了计算技术以在实践中有效近似。初步实验结果揭示了这种新的自然梯度变体的潜力。
While natural gradients have been widely studied from both theoretical and empirical perspectives, we argue that some fundamental theoretical issues regarding the existence of gradients in infinite dimensional function spaces remain underexplored. We address these issues by providing a geometric perspective and mathematical framework for studying natural gradient that is more complete and rigorous than existing studies. Our results also establish new connections between natural gradients and RKHS theory, and specifically to the Neural Tangent Kernel (NTK). Based on our theoretical framework, we derive a new family of natural gradients induced by Sobolev metrics and develop computational techniques for efficient approximation in practice. Preliminary experimental results reveal the potential of this new natural gradient variant.
