论文标题
用于预测性编码网络的稳定，快速且全自动学习算法
A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks
论文作者
论文摘要
预测性编码网络是具有神经科学启发的模型，其根源在贝叶斯统计和神经科学中。但是，培训此类模型效率很低且不稳定。在这项工作中，我们展示了如何简单地更改突触权重的更新规则的时间调度，从而导致一种算法比原始的算法更有效和稳定，并且在收敛方面具有理论上的保证。我们称为增量预测编码（IPC）的算法在生物学上比原始的算法更合理，因为它是完全自动的。在一组广泛的实验中，我们表明IPC在大量基准测试基准上的原始配方不断地表现出色，并且在测试准确性，效率和收敛方面，相对于大量的超参数，对有条件和掩盖语言模型的培训都比训练条件和蒙版语言模型的培训更好。
Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters.
