DreamFusion: Text-to-3D using 2D Diffusion

DreamFusion performs text-to-3D synthesis without any 3D training data or modifications to an image diffusion model. Because large labeled 3D datasets and efficient 3D denoisers do not exist, the method instead leverages a pretrained 2D text-to-image diffusion model as a prior. It introduces a loss based on probability density distillation and, in a DeepDream-like optimization procedure, uses gradient descent to optimize a randomly initialized Neural Radiance Field so that its 2D renderings from random viewing angles achieve low loss under the diffusion prior.

The resulting 3D model, produced from a text prompt, can be viewed from any angle, relit under arbitrary illumination, and composited into any 3D environment. By circumventing the need for 3D data and requiring no changes to the underlying image diffusion model, DreamFusion demonstrated the effectiveness of pretrained 2D image diffusion models as priors for 3D generation, an influential result for text-driven 3D content creation.