DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption by
Combining 3D GANs and Diffusion Priors



Text-guided domain adaption and generation of 3D-aware portraits find many applications in various fields. However, due to the lack of training data and the challenges in handling the high variety of geometry and appearance, the existing methods for these tasks suffer from issues like inflexibility, instability, and low fidelity. In this paper, we propose a novel framework DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by combining 3D GANs and diffusion priors. Specifically, we integrate the pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion models. The former provides a strong foundation for stable and high-quality avatar generation from text. And the diffusion models in turn offer powerful priors and guide the 3D generator finetuning with informative direction to achieve flexible and efficient text-guided domain adaption. To enhance the diversity in domain adaption and the generation capability in text-to-avatar, we introduce the relative distance loss and case-specific learnable triplane respectively. Besides, we design a progressive texture refinement module to improve the texture quality for both tasks above. Extensive experiments demonstrate that the proposed framework achieves excellent results in both domain adaption and text-to-avatar tasks, outperforming existing methods in terms of generation quality and efficiency.


DiffusionGAN3D is a novel two-stage framework, which aims to boost the performance of 3D domain adaption and text-to-avatar tasks by combining 3D generative models and diffusion priors, as shown in the figure below.



  • Text-guided 3D stylization (face).
  • Sampled images:
    "pixar style" "lego head" "zombie"
    "plaster statue" "oil painting" "caricature style"

    + inversion:
    "pixar style" "plaster statue"
    "zombie" "caricature style"

  • 3D local editing (face).
  • real image inversion "blue eyes"
    real image inversion "pink hair"

  • Text-to-Avatar (head).
  • "Batman" "Hulk" "Obama" "a woman with brown short hair"