论文标题
在评估协议中解决分布外检测的随机性
Addressing Randomness in Evaluation Protocols for Out-of-Distribution Detection
论文作者
论文摘要
当面对不源于训练分布的意见时，用于分类的深层神经网络会表现得不可预测。这激发了分布外检测（OOD）机制。通常缺乏有关分布数据数据的先前信息，这使得难以看见的数据的检测方法的性能估计。几种当代评估协议基于开放式仿真，该模拟平均具有五个数据集的五个合成随机分裂的平均性能，将数据集分为分布式和分离外样品中。但是，可能的拆分数量可能会大得多，并且已知深神经网络的性能会显着波动，具体取决于随机变化的不同来源。我们从经验上证明，当前的协议可能无法对OOD方法的预期性能提供可靠的估计。通过将此评估作为一个随机过程，我们将开放式模拟的概念推广，并建议使用解决随机性的蒙特卡洛方法估算OOD方法的性能。
Deep Neural Networks for classification behave unpredictably when confronted with inputs not stemming from the training distribution. This motivates out-of-distribution detection (OOD) mechanisms. The usual lack of prior information on out-of-distribution data renders the performance estimation of detection approaches on unseen data difficult. Several contemporary evaluation protocols are based on open set simulations, which average the performance over up to five synthetic random splits of a dataset into in- and out-of-distribution samples. However, the number of possible splits may be much larger, and the performance of Deep Neural Networks is known to fluctuate significantly depending on different sources of random variation. We empirically demonstrate that current protocols may fail to provide reliable estimates of the expected performance of OOD methods. By casting this evaluation as a random process, we generalize the concept of open set simulations and propose to estimate the performance of OOD methods using a Monte Carlo approach that addresses the randomness.
