论文标题
差异化私人优化器可以学习对抗性强大的模型
Differentially Private Optimizers Can Learn Adversarially Robust Models
论文作者
论文摘要
机器学习模型在各种领域都发挥了作用,并引起了安全和隐私社区的越来越多的关注。一个重要但令人担忧的问题是:在差异隐私(DP)约束下训练模型是否会对其对抗性鲁棒性产生不利影响?尽管以前的作品假定隐私是以较差的稳健性来实现的,但我们进行了第一个理论分析,以表明DP模型确实可以是健壮和准确的,甚至比其自然受过训练的非私有私人对应物更健壮。我们观察到影响隐私性 - 准确性权准的权衡的三个关键因素:(1)DP优化器的超参数至关重要; (2)对公共数据进行预培训可显着降低准确性和鲁棒性下降; (3)DP优化器的选择会有所不同。通过正确设定了这些因素,我们在$ L_2(0.5)$攻击下达到90 \%的自然精度(比非私人模型的$+9 \%$)($+9 \%$),而69 \%\%\%的稳健精度($+16 \%$ bed vant nontr-Provate型号)在Pre-training simclrv2下使用$ l_ \ l _ $ l_ \ l_的$ 10(4/ $ε= 2 $。实际上,我们从理论和经验上都表明,DP模型是精确度折衷方案的最佳选择。从经验上讲,在各种数据集和模型中一直观察到DP模型的鲁棒性。我们认为,我们令人鼓舞的结果是朝着私人和强大的培训模型迈出的重要一步。
Machine learning models have shone in a variety of domains and attracted increasing attention from both the security and the privacy communities. One important yet worrying question is: Will training models under the differential privacy (DP) constraint have an unfavorable impact on their adversarial robustness? While previous works have postulated that privacy comes at the cost of worse robustness, we give the first theoretical analysis to show that DP models can indeed be robust and accurate, even sometimes more robust than their naturally-trained non-private counterparts. We observe three key factors that influence the privacy-robustness-accuracy tradeoff: (1) hyper-parameters for DP optimizers are critical; (2) pre-training on public data significantly mitigates the accuracy and robustness drop; (3) choice of DP optimizers makes a difference. With these factors set properly, we achieve 90\% natural accuracy, 72\% robust accuracy ($+9\%$ than the non-private model) under $l_2(0.5)$ attack, and 69\% robust accuracy ($+16\%$ than the non-private model) with pre-trained SimCLRv2 model under $l_\infty(4/255)$ attack on CIFAR10 with $ε=2$. In fact, we show both theoretically and empirically that DP models are Pareto optimal on the accuracy-robustness tradeoff. Empirically, the robustness of DP models is consistently observed across various datasets and models. We believe our encouraging results are a significant step towards training models that are private as well as robust.