What You See is What You GAN (In 3D)
So essentially,
We can now synthesize high-resolution 3D geometry from 2D.
Paper: What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs(27 Pages)
Researchers from the NVIDIA and University of California, San Diego are interested in developing better performant 2D to 3D models.
Hmm..What’s the background?
3D representations, specifically Neural Radiance Fields (NeRF), are a common 3D backbone for use in various 3D Generative Adversarial Networks (GANs).
The computational and memory cost of rendering such images is prohibitive. To alleviate these costs, this study focuses on accelerating the rendering process for 3D GANs
Ok, So what is proposed in the research paper?
The main proposal is a novel sampling method based on learning-based samplers. These samplers efficiently predict high-resolution discrete distributions in the image domain. The sampling strategy involves learning an intermediate geometry representation that is then transformed into an opacity value.
The results of this research show that the proposed method can achieve state-of-the-art 3D geometric quality on FFHQ and AFHQ datasets, setting a new standard for unsupervised learning of 3D shapes in 3D GANs. Additionally, the method significantly accelerates the rendering process, making it more feasible to train 3D GANs with high-resolution images
And what’s next?
The researchers acknowledge that their method may produce artifacts such as dents in the presence of specularities and cannot handle transparent objects such as lenses well. They also note that future work could incorporate more advanced material formulations and surface normal regularization to address these limitations.
So essentially,
We can now synthesize high-resolution 3D geometry from 2D.