论文标题
2D甘斯知道3D形状吗？ 2D图像gan的无监督3D形状重建
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
论文作者
论文摘要
自然图像是2D图像平面上3D对象的投影。虽然诸如gans之类的最新2D生成模型在建模自然图像歧管时表现出了前所未有的质量，但尚不清楚它们是否暗中捕获了基本的3D对象结构。如果是这样，我们如何利用此类知识来恢复图像中对象的3D形状？为了回答这些问题，在这项工作中，我们提出了第一次尝试从仅在RGB图像上训练的现成的2D GAN直接挖掘3D几何提示。通过我们的调查，我们发现这种预训练的GAN确实包含了丰富的3D知识，因此可以用无监督的方式从单个2D图像中恢复3D形状。我们框架的核心是一种迭代策略，它探索和利用了GAN图像歧管中的各种观点和照明变化。该框架不需要2D关键点或3D注释，也不需要对物体形状的强烈假设（例如形状是对称的），但是它成功地恢复了人脸，猫，汽车和建筑物的高精度3D形状。恢复的3D形状立即允许高质量的图像编辑，例如重新定义和对象旋转。与3D形状重建和面部旋转中的先前方法相比，我们定量证明了方法的有效性。我们的代码可在https://github.com/xingangpan/gan2shape上找到。
Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.
