3D人脸重构论文汇总【PV3D\EG3D\AvatarGen\Face2FaceRHO\RODIN\DCFace\SadTalker\NeuFace\Next3D\SCULPT\HumanLiff等】

目录

《Face2FaceRHO: Real-Time High-Resolution One-Shot Face Reenactment》

《RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion》

《DCFace: Synthetic Face Generation with Dual Condition Diffusion Model》

《Graphics Capsule: Learning Hierarchical 3D Face Representations from 2D Images》

《SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation》

《Learning 3D-aware Image Synthesis with Unknown Pose Distribution》

《NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images》

《Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars》

《SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes》】

《HumanLiff: Layer-wise 3D Human Generation with Diffusion Model》

《PV3D-A 3D GENERATIVE MODEL FOR PORTRAIT》 2023 ICLR

《EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks》  2022CVPR

《AvatarGen: A 3D Generative Model for Animatable Human Avatars》 2022ECCV

《Gram: Generative radiance manifolds for 3d-aware image generation》

Stylesdf: High-resolution 3d-consistent image and geometry generation.

Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images.

A high-resolution 3d-aware generative model.

 Periodic implicit generative adversarial networks for 3d-aware image synthesis 

Alias-free generative adversarial networks.

其他相关的碎碎念

---

《Face2FaceRHO: Real-Time High-Resolution One-Shot Face Reenactment》

2022 ECCV 网易AI 基于单幅图片的实时高分辨率人脸重演算法

Face2FaceRHOhttps://github.com/NetEase-GameAI/Face2FaceRHOFace2FaceRHO论文精读https://blog.csdn.net/lgzlgz3102/article/details/125955000?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522169286291116800185816933%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=169286291116800185816933&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~times_rank-29-125955000-null-null.142%5Ev93%5EchatsearchT3_1&utm_term=%E4%BA%BA%E8%84%B8%E7%94%9F%E6%88%903D&spm=1018.2226.3001.4187

《RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion》

2023 CVPR 微软提出RODIN（ Roll-out Diffusion Network ）

RODINhttps://blog.csdn.net/amusi1994/article/details/129645915?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522169286408716800213014490%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=169286408716800213014490&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~times_rank-19-129645915-null-null.142%5Ev93%5EchatsearchT3_1&utm_term=%E4%BA%BA%E8%84%B8%E7%94%9F%E6%88%903D&spm=1018.2226.3001.4187

《DCFace: Synthetic Face Generation with Dual Condition Diffusion Model》

DCFacehttps://arxiv.org/abs/2304.07060v1

1）方向：人脸识别技术

2）应用：生成用于训练人脸识别模型的合成数据集

3）背景：生成合成数据集需要考虑多个因素，如姿势、光照、表情、年龄和遮挡等，以模拟真实图像的条件分布。以往的研究主要使用GAN或3D模型生成合成数据集。

4）方法：本文提出一种基于扩散模型的双重条件人脸生成器（DCFace），通过结合主体外观（ID）和外部因素（风格）条件来控制类内和类间变化。作者使用了新颖的Patch-wise风格提取器和Time-step依赖的ID损失，使DCFace能够在不同风格下精确控制同一主体的人脸图像生成。

5）结果：使用DCFace生成的合成数据集训练的人脸识别模型在LFW、CFP-FP、CPLFW、AgeDB和CALFW等5个测试数据集中，平均比以前的方法提高了6.11％的验证准确性。代码可在https://github.com/mk-minchul/dcface上获得。
 

《Graphics Capsule: Learning Hierarchical 3D Face Representations from 2D Images》

构建对象层次结构，对于人脑视觉过程非常重要。以前的研究已经成功地采用胶囊网络将digits和面部分解为部件，以无监督的方式研究神经网络中类似的感知机制。然而，它们的描述仅限于2D空间，限制了它们模仿人类固有的3D感知能力。

本文提出一种逆图形胶囊网络（Inverse Graphics Capsule Network，IGC-Net），用于从大规模未标记图像中学习分层3D人脸表示。IGC-Net的核心是一种新型胶囊，名为图形胶囊，它以计算机图形（CG）中的可解释参数表示3D图元，包括深度、反照率和3D姿态。

具体而言，IGC-Net首先将对象分解成一组语义一致的部分级描述，然后将它们组装成对象级描述以构建层次结构。学到的图形胶囊揭示了面向视觉感知的神经网络如何将面孔理解为3D模型的层次结构。此外，发现的部件可以部署到无监督的人脸分割任务中，以评估方法的语义一致性。此外，具有显式物理含义的部分级描述为原本在黑匣子中运行的面部分析提供了见解，例如形状和纹理对于面部识别的重要性。CelebA， BP4D 和 Multi-PIE 上的实验展示了IGC-Net。

《SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation》

通过人脸图像和一段语音音频生成说话头像视频（talking head）仍然存在许多挑战。即，不自然的头部运动，扭曲的表情和身份修改。这些问题主要是因为从耦合的2D运动场中学习。另一方面，显式使用3D信息也遇到了表达僵硬和非连贯视频的问题。

提出了SadTalker，它从音频中生成3D运动系数（头部姿态，表情），并隐式调制新的3D感知脸部渲染以实现说话头生成。为了学习真实的运动系数，显式地建立了音频与不同类型的运动系数之间的连接。具体来说，提出了ExpNet，通过提取系数和3D渲染面部来从音频中学习准确的面部表情。至于头部姿势，设计了一个基于条件VAE的PoseVAE，以生成不同风格的头部运动。最后，将生成的3D运动系数映射到所提议的面部渲染的无监督3D关键点空间，并合成最终视频。

大量实验证明方法在运动和视频质量方面的优越性：https://sadtalker.github.io/
 

《Learning 3D-aware Image Synthesis with Unknown Pose Distribution》

现有的3D感知图像合成方法很大程度上依赖于在训练集上预先估计的3D姿态分布。如果估计不准确，可能会误导模型去学习错误的几何信息。这项工作提出了PoF3D，它释放了生成辐射场对3D姿态先验的需求。

首先为生成器配置一种有效的姿态学习器，能够从一个潜在的编码中推断出姿态，以自动近似底层真实的姿态分布。接着，分配给鉴别器一个在生成器的监督下学习姿态分布的任务，并以预测的姿态作为条件区分实际和合成图像。姿态自由生成器和姿态感知鉴别器以对抗性的方式共同训练。

一系列数据集上的结果证实，方法在图像质量和几何质量方面的性能与最先进的方法相当。PoF3D首次证明了在不使用3D姿态先验的情况下学习高质量的3D感知图像合成的可行性。项目页面：https://vivianszf.github.io/pof3d/
 

《NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images》

从多视角图像进行真实感人脸渲染（face rendering），有利于各种计算机视觉和图形应用任务。然而，由于人脸具有复杂的空间变化的反射特性和几何特征，因此在当前的研究中，恢复3D人脸表示仍然具有挑战性。

本文提出了一种新的3D人脸渲染模型，即NeuFace，通过神经渲染技术学习精确和物理意义上有意义的底层3D表示。它自然地将神经BRDFs融入到基于物理的渲染中，以协作方式捕获复杂的面部几何和外观线索。具体来说，引入了一种近似的BRDF积分和一个简单而新的低秩先验，有效地降低了人脸BRDF的模糊性并提高了性能。大量实验证明了NeuFace在人脸渲染方面的优越性，以及在常见物体上的良好泛化能力。

代码已在NeuFace上发布：https://github.com/aejion/NeuFace
 

《Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars》

3D感知生成对抗网络（GANs）仅使用单视角2D图像合成高保真度和多视角一致的面部图像。为实现对面部属性的细粒度控制，近期的研究努力将3D形变人脸模型（3D Morphable Face Model，3DMM）纳入生成辐射场的显式或隐式描述中。显式方法提供细粒度的表达控制，但无法处理由头发和配饰引起的拓扑变化，而隐式方法可以建模各种拓扑，但由于不受约束的变形场，其适用范围有限。

提出了一种新的3D GAN框架，用于从无结构的2D图像中无监督学习生成高质量且具备3D一致性的面部形象。为实现变形精度和拓扑灵活性，提出了一种名为生成纹理光栅化三角面的3D表示。所提出的表示在参数化网格模板之上学习生成神经纹理，然后通过光栅化将它们投影到三个正交视角的特征平面上，形成一个三角面的体积渲染。这样，结合了网格引导的显式变形的细粒度表达控制和隐式体积表示的灵活性。进一步提出了用于建模不受3DMM影响的嘴部特定模块。

方法通过广泛的实验展示了最先进的3D感知合成质量和动画能力。此外，作为3D先验的可驱动3D表示在单样本人脸虚拟化和3D感知风格化等多个应用中起到了推动作用。
 

《SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes》】

SCULPT（2023）http://SCULPT

We present SCULPT, a novel 3D generative model for clothed and textured 3D meshes of humans. Specifically, we devise a deep neural network that learns to represent the geometry and appearance distribution of clothed human bodies. Training such a model is challenging, as datasets of textured 3D meshes for humans are limited in size and accessibility. Our key observation is that there exist medium-sized 3D scan datasets like CAPE, as well as large-scale 2D image datasets of clothed humans and multiple appearances can be mapped to a single geometry. To effectively learn from the two data modalities, we propose an unpaired learning procedure for pose-dependent clothed and textured human meshes. Specifically, we learn a pose-dependent geometry space from 3D scan data. We represent this as per vertex displacements w.r.t. the SMPL model. Next, we train a geometry conditioned texture generator in an unsupervised way using the 2D image data. We use intermediate activations of the learned geometry model to condition our texture generator. To alleviate entanglement between pose and clothing type, and pose and clothing appearance, we condition both the texture and geometry generators with attribute labels such as clothing types for the geometry, and clothing colors for the texture generator. We automatically generated these conditioning labels for the 2D images based on the visual question answering model BLIP and CLIP. We validate our method on the SCULPT dataset, and compare to state-of-the-art 3D generative models for clothed human bodies. We will release the codebase for research purposes.

这篇文章是3D虚拟服装生成的，不是人脸3D的，就在这里暂存一下吧

《HumanLiff: Layer-wise 3D Human Generation with Diffusion Model》

[2308.09712] HumanLiff: Layer-wise 3D Human Generation with Diffusion Model (arxiv.org)https://arxiv.org/abs/2308.09712也不是人脸生成，讲的是三维人体生成模型，使用了Diffusion，后面可以细细看

《PV3D-A 3D GENERATIVE MODEL FOR PORTRAIT》 2023 ICLR

PV3D 论文精读https://blog.csdn.net/qq_53826699/article/details/132391072?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22132391072%22%2C%22source%22%3A%22qq_53826699%22%7D

首个3D人像视频生成模型！仅需1张2D人像，眨眼、口型都能改变https://blog.csdn.net/amusi1994/article/details/129134765?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522169286291116800185816933%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=169286291116800185816933&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~times_rank-26-129134765-null-null.142%5Ev93%5EchatsearchT3_1&utm_term=%E4%BA%BA%E8%84%B8%E7%94%9F%E6%88%903D&spm=1018.2226.3001.4187

《EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks》  2022CVPR

EG3D论文精读https://blog.csdn.net/qq_53826699/article/details/132459041?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22132459041%22%2C%22source%22%3A%22qq_53826699%22%7D

《AvatarGen: A 3D Generative Model for Animatable Human Avatars》 2022ECCV

【2022-ECCV（European Conference on Computer Vision）】

AvatarGen（ECCV 2022）https://link.springer.com/chapter/10.1007/978-3-031-25066-8_39

Unsupervised generation of clothed virtual humans with various appearance and animatable poses is important for creating 3D human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to synthesize high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only non-rigid human generation with diverse appearance but also full control over poses and viewpoints, while only requiring 2D images for training. Specifically, it extends the recent 3D GANs to clothed human generation by utilizing a coarse human body model as a proxy to warp the observation space into a standard avatar under a canonical space. To model non-rigid dynamics, it introduces a deformation network to learn pose-dependent deformations in the canonical space. To improve geometry quality of the generated human avatars, it leverages signed distance field as geometric representation, which allows more direct regularization from the body model on the geometry learning. Benefiting from these designs, our method can generate animatable human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs. Furthermore, it is competent for many applications, e.g., single-view reconstruction, reanimation, and text-guided synthesis.

无监督生成具有各种外观和动画姿势的穿衣虚拟人对于创建三维人体化身和其他 AR/VR 应用非常重要。现有的方法要么仅限于刚性物体建模，要么不具备生成性，因此无法合成高质量的虚拟人并为其制作动画。在这项工作中，我们提出了 AvatarGen，这是第一种不仅能生成具有不同外观的非刚性人形，还能完全控制姿势和视角的方法，同时只需要二维图像进行训练。具体来说，该方法利用粗糙人体模型作为代理，将观测空间扭曲为标准空间下的标准头像，从而将最近的三维 GAN 扩展到穿衣人体生成。为了建立非刚性动态模型，它引入了一个变形网络来学习典型空间中与姿势相关的变形。为了提高生成的人体头像的几何质量，它利用签名距离场作为几何表示，这使得人体模型对几何学习的正则化更加直接。得益于这些设计，我们的方法可以生成具有高质量外观和几何建模的可动画人类头像，大大优于之前的 3D GAN。此外，它还适用于多种应用，例如单视角重建、重塑和文本引导合成。

从references扒出来的，还没读

《Gram: Generative radiance manifolds for 3d-aware image generation》

In: CVPR (2022)

Stylesdf: High-resolution 3d-consistent image and geometry generation.

In: CVPR (2022)

Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images.

In: CVPR (2022)

A high-resolution 3d-aware generative model.

In: CVPR (2022)

 Periodic implicit generative adversarial networks for 3d-aware image synthesis 

CVPR (2021)

Alias-free generative adversarial networks.

In: NeurIPS (2021)

其他相关的碎碎念

2021 华为&上交 CIPS-3Dhttps://blog.csdn.net/Yong_Qi2015/article/details/121112734?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522169286408716800213014490%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=169286408716800213014490&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~times_rank-20-121112734-null-null.142%5Ev93%5EchatsearchT3_1&utm_term=%E4%BA%BA%E8%84%B8%E7%94%9F%E6%88%903D&spm=1018.2226.3001.4187

【计算机视觉|人脸建模】3D人脸重建基础知识（入门）https://blog.csdn.net/I_am_Tony_Stark/article/details/132011344?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522169286245316800197012555%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=169286245316800197012555&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~times_rank-1-132011344-null-null.142%5Ev93%5EchatsearchT3_1&utm_term=%E4%BA%BA%E8%84%B8%E7%94%9F%E6%88%903D&spm=1018.2226.3001.4187

人脸相关科研笔记宝藏博主https://blog.csdn.net/qq_45934285/category_12316953.html