论文标题
跨度：随机预测的近似牛顿方法
SPAN: A Stochastic Projected Approximate Newton Method
论文作者
论文摘要
二阶优化方法具有理想的收敛属性。但是，确切的牛顿方法需要针对黑森州及其倒数的昂贵计算。在本文中，我们提出了跨度，这是一种新颖而快速的牛顿方法。 SPAN通过低级别近似和随机的黑森 - 矢量产物计算Hessian矩阵的倒数。我们在多个基准数据集上进行的实验表明，跨越范围的隔离壁锁定时间的范围优于现有的一阶和二阶优化方法。此外，我们提供了对触觉复杂性，近似误差和收敛速率的理论分析。理论分析和实验结果都表明，我们提出的方法在收敛速率和触电效率之间取得了更好的权衡。
Second-order optimization methods have desirable convergence properties. However, the exact Newton method requires expensive computation for the Hessian and its inverse. In this paper, we propose SPAN, a novel approximate and fast Newton method. SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products. Our experiments on multiple benchmark datasets demonstrate that SPAN outperforms existing first-order and second-order optimization methods in terms of the convergence wall-clock time. Furthermore, we provide a theoretical analysis of the per-iteration complexity, the approximation error, and the convergence rate. Both the theoretical analysis and experimental results show that our proposed method achieves a better trade-off between the convergence rate and the per-iteration efficiency.
