Ink to Incarnate: Consistent 3D Animals with YouDream 🐕🐎🐃🐘
So essentially,
YOUDREAM Auto-Generates Perfect 3D Creatures from Text!
Paper: YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals (23 Pages)
Researchers from University of Texas at Austin and University of Massachusetts Amherst are interesting in generating controllable, consistent and proportionate 3D animals.
Hmm..What’s the background?
Existing text-to-3D generation methods struggle to create anatomically correct and customizable animal models. They often rely on text or images alone, which limits creative control and can result in geometric inconsistencies, such as the "Janus-head" problem. Existing parametric models for animals, like SMAL and MagicPony, suffer from diversity issues and are not suitable for representing the vastness of the animal kingdom.
Ok, So what is proposed in the research paper?
The paper introduces a multi-agent LLM system to automatically generate 3D poses for common animals based on textual descriptions and a small library of predefined poses
This system helps circumvent the need for large and diverse 3D animal pose datasets, which are currently unavailable
A key component of the method is the training of a ControlNet specifically for tetrapod animals. This network learns to generate images adhering to 2D poses derived from a 3D pose prior. The training utilizes 2D animal pose datasets and maps 2D poses to corresponding images, enabling the generation of multi-view image samples consistent with the 3D pose
YOUDREAM includes a user-friendly tool that enables the creation and editing of 3D poses. This tool facilitates the modification, addition, or deletion of joints and bones to design custom animal skeletons. Additionally, it provides an automatic shape initializer that generates an initial 3D shape based on the designed skeleton, simplifying the NeRF initialization process
Source: https://huggingface.co/papers/2406.16273
What’s next?
The authors acknowledge that although YOUDREAM generates anatomically sound animals, the visual fidelity of textures and overall sharpness could be enhanced. Future research could focus on developing more efficient methods for generating controlled animations using the YOUDREAM framework.
As YOUDREAM builds upon the Stable Diffusion model, it inherits any biases present in the training data of Stable Diffusion. The authors emphasize the importance of addressing these biases to ensure responsible and fair use of the technology. Future efforts should focus on identifying and mitigating potential biases in the generated 3D animals.
So essentially,
YOUDREAM Auto-Generates Perfect 3D Creatures from Text!