论文标题
添加噪声的好处在具有界限的构图中
Benefits of Additive Noise in Composing Classes with Bounded Capacity
论文作者
论文摘要
我们观察到，给定两种（兼容的）函数类别$ \ MATHCAL {F} $和$ \ MATHCAL {H} $，其容量很小，其均匀覆盖的数字衡量，组成类$ \ MATHCAL {H} \ MATHCAL {H} \ MATHCAL {f CIRCE \ MATHCAL {F}的容量可能会变得更加宽容，甚至是无孔的。然后，我们证明，在用$ \ Mathcal {h} $组成$ \ Mathcal {f} $的输出中，添加少量高斯噪声可以有效地控制$ \ Mathcal {H} \ Circ \ Mathcal {F} $的容量。为了证明我们的结果，我们定义了有关总变化和瓦斯坦河距离的均匀数量的均匀数量的新概念。我们将结果实例化，以实现多层Sigmoid神经网络。 MNIST数据集的初步经验结果表明，在现有统一界限上改善所需的噪声量在数值上可以忽略不计（即，元素的I.I.D. I.I.D.高斯噪声，具有标准偏差$ 10^{ - 240} $）。源代码可从https://github.com/fathollahpour/composition_noise获得。
We observe that given two (compatible) classes of functions $\mathcal{F}$ and $\mathcal{H}$ with small capacity as measured by their uniform covering numbers, the capacity of the composition class $\mathcal{H} \circ \mathcal{F}$ can become prohibitively large or even unbounded. We then show that adding a small amount of Gaussian noise to the output of $\mathcal{F}$ before composing it with $\mathcal{H}$ can effectively control the capacity of $\mathcal{H} \circ \mathcal{F}$, offering a general recipe for modular design. To prove our results, we define new notions of uniform covering number of random functions with respect to the total variation and Wasserstein distances. We instantiate our results for the case of multi-layer sigmoid neural networks. Preliminary empirical results on MNIST dataset indicate that the amount of noise required to improve over existing uniform bounds can be numerically negligible (i.e., element-wise i.i.d. Gaussian noise with standard deviation $10^{-240}$). The source codes are available at https://github.com/fathollahpour/composition_noise.
