Paper

Video

Code

Demo (Coming soon)

Abstract

Text-guided domain adaption and generation of 3D-aware portraits find many applications in various fields. However, due to the lack of training data and the challenges in handling the high variety of geometry and appearance, the existing methods for these tasks suffer from issues like inflexibility, instability, and low fidelity. In this paper, we propose a novel framework DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by combining 3D GANs and diffusion priors. Specifically, we integrate the pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion models. The former provides a strong foundation for stable and high-quality avatar generation from text. And the diffusion models in turn offer powerful priors and guide the 3D generator finetuning with informative direction to achieve flexible and efficient text-guided domain adaption. To enhance the diversity in domain adaption and the generation capability in text-to-avatar, we introduce the relative distance loss and case-specific learnable triplane respectively. Besides, we design a progressive texture refinement module to improve the texture quality for both tasks above. Extensive experiments demonstrate that the proposed framework achieves excellent results in both domain adaption and text-to-avatar tasks, outperforming existing methods in terms of generation quality and efficiency.

Architecture

DiffusionGAN3D is a novel two-stage framework, which aims to boost the performance of 3D domain adaption and text-to-avatar tasks by combining 3D generative models and diffusion priors, as shown in the figure below.

Results

Text-guided 3D stylization (face).

Sampled images:

"pixar style"	"lego head"	"zombie"
"plaster statue"	"oil painting"	"caricature style"

+ inversion:

	"pixar style"		"plaster statue"
	"zombie"		"caricature style"

3D local editing (face).

real image	inversion	"blue eyes"
real image	inversion	"pink hair"

Text-to-Avatar (head).

"Batman"	"Hulk"	"Obama"	"a woman with brown short hair"

Citation

@misc{lei2023diffusiongan3d, title={DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption by Combining 3D GANs and Diffusion Priors}, author={Biwen Lei and Kai Yu and Mengyang Feng and Miaomiao Cui and Xuansong Xie}, year={2023}, eprint={2312.16837}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Acknowledgements