论文标题
Adan：自适应Nesterov动量算法，用于更快地优化深层模型
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
论文作者
论文摘要
在深度学习中，不同种类的深网通常需要不同的优化器，这些优化器必须在多次试验后选择，从而使培训过程效率低下。为了缓解此问题并持续提高深层网络的模型训练速度，我们提出了Nesterov动量算法，Adan简称Adan。 Adan首先重新制定了Nesterov加速度，以开发新的Nesterov动量估计（NME）方法，该方法避免了外推点上计算梯度的额外开销。然后，Adan采用NME来估计梯度在自适应梯度算法中的一阶和二阶时刻，以进行收敛加速。此外，我们证明Adan在$ \ Mathcal {o}（ε^{ - 3.5}）$ $ε$ - $ - $ - $ - $ - $ - $ \ Mathcal {o}（ε^{ - 3.5}）$随机梯度的复杂性（例如，深度学习问题）（例如，深度学习问题），与最著名的下限相匹配。广泛的实验结果表明，Adan始终超过有关视觉，语言和RL任务的相应SOTA优化器，并为许多流行的网络和框架设置了新的SOTA，例如Resnet，Convnext，Vit，Vit，Vit，Swin，Mae，Mae，Detr，Gpt-2，Gpt-2，gpt-2，Transformer-XL和Bert和Bert。更令人惊讶的是，Adan可以使用SOTA优化器的一半培训成本（时期）来在VIT，GPT-2，MAE等上取得更高或可比的性能，并且还显示出对大型Minibatch尺寸（例如1K至32K）的宽敞宽容。代码可在https://github.com/sail-sg/adan上发布，并已用于多个流行的深度学习框架或项目。
In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then, Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an $ε$-approximate first-order stationary point within $\mathcal{O}(ε^{-3.5})$ stochastic gradient complexity on the non-convex stochastic problems (e.g., deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, e.g., ResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc., and also shows great tolerance to a large range of minibatch size, e.g., from 1k to 32k. Code is released at https://github.com/sail-sg/Adan, and has been used in multiple popular deep learning frameworks or projects.
