论文标题
高斯混合物的熵近似的理论误差分析
Theoretical Error Analysis of Entropy Approximation for Gaussian Mixtures
论文作者
论文摘要
高斯混合物分布通常用于代表一般概率分布。尽管将高斯混合物用于不确定性估计很重要，但无法通过分析计算高斯混合物的熵。在本文中，我们研究了用混合系数的单峰高斯分布的熵之和表示的近似熵。无论尺寸如何，这种近似都很容易在分析上计算，但是缺乏理论保证。我们理论上分析了真实熵和近似熵之间的近似误差，以揭示此近似何时有效工作。该误差本质上是由高斯混合物的每个高斯组件分开多远来控制的。为了衡量这种分离，我们介绍了平均值之间的距离与高斯混合物的每个高斯组分的方差之和之和的比率，并且我们揭示了由于比率趋于无穷大，误差会收敛到零。此外，概率估计表明，这种收敛情况更可能发生在较高维度的空间中。因此，我们的结果提供了保证，这种近似值对于高维问题（例如涉及大量参数的神经网络）效果很好。
Gaussian mixture distributions are commonly employed to represent general probability distributions. Despite the importance of using Gaussian mixtures for uncertainty estimation, the entropy of a Gaussian mixture cannot be calculated analytically. In this paper, we study the approximate entropy represented as the sum of the entropies of unimodal Gaussian distributions with mixing coefficients. This approximation is easy to calculate analytically regardless of dimension, but there is a lack of theoretical guarantees. We theoretically analyze the approximation error between the true and the approximate entropy to reveal when this approximation works effectively. This error is essentially controlled by how far apart each Gaussian component of the Gaussian mixture is. To measure such separation, we introduce the ratios of the distances between the means to the sum of the variances of each Gaussian component of the Gaussian mixture, and we reveal that the error converges to zero as the ratios tend to infinity. In addition, the probabilistic estimate indicates that this convergence situation is more likely to occur in higher-dimensional spaces. Therefore, our results provide a guarantee that this approximation works well for high-dimensional problems, such as neural networks that involve a large number of parameters.
