论文标题
广泛的米尼马密度假设和探索探索学习率计划
Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
论文作者
论文摘要
几篇论文认为，宽阔的最小值比狭窄的最小值更好。在本文中，通过详细的实验不仅证实了广泛的最小值的概括特性，我们还为新假设提供了经验证据，即宽度最小值的密度可能低于狭窄的最小值密度。此外，在这一假设的推动下，我们设计了一个新颖的探索探索学习率计划。在各种图像和自然语言数据集上，与原始手动学习率基准相比，我们表明我们的探索探索时间表可以使用原始培训预算高达0.84％的绝对准确性，或者最多可减少57％的培训时间，同时达到原始报告的准确性。例如，我们通过仅修改高性能模型的学习率时间表来实现IWSLT'14（DE-EN）数据集的最新精度（SOTA）。
Several papers argue that wide minima generalize better than narrow minima. In this paper, through detailed experiments that not only corroborate the generalization properties of wide minima, we also provide empirical evidence for a new hypothesis that the density of wide minima is likely lower than the density of narrow minima. Further, motivated by this hypothesis, we design a novel explore-exploit learning rate schedule. On a variety of image and natural language datasets, compared to their original hand-tuned learning rate baselines, we show that our explore-exploit schedule can result in either up to 0.84% higher absolute accuracy using the original training budget or up to 57% reduced training time while achieving the original reported accuracy. For example, we achieve state-of-the-art (SOTA) accuracy for IWSLT'14 (DE-EN) dataset by just modifying the learning rate schedule of a high performing model.
