论文标题
使用软注意机制的自适应物理信息感染的神经网络
Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
论文作者
论文摘要
物理知识的神经网络（PINN）最近出现了，是深层神经网络在非线性偏微分方程（PDES）的数值解决方案中的有前途应用。但是，已经认识到需要自适应程序来迫使神经网络准确地拟合顽固的斑点“僵硬” PDES。在本文中，我们提出了一种从根本上进行自适应训练Pinn的新方法，在该方法中，适应权重完全训练并分别应用于每个训练点，因此神经网络自动学习了解决方案的哪些区域很难，并且被迫专注于它们。自我适应权重指定了软乘式软注意面膜，这让人想起计算机视觉中使用的类似机制。这些SA-Pinn的基本思想是随着相应的损失的增加，权重增加，这是通过训练网络同时最大程度地减少损失并最大化权重来完成的。此外，我们还展示了如何使用高斯工艺回归来构建自适应权重的连续图，这允许在常规梯度下降不足以产生准确的溶液中使用随机梯度下降。 Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms.在具有几个线性和非线性基准问题的数值实验中，SA-Pinn在L2误差中优于其他最先进的Pinn算法，同时使用较小数量的训练时期。
Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.
