Da-Vinci: Self-Supervised Generative Model For Game Character Synthesis
Contributors
Murali, Ramanan, Ajay, Ganesh
Overview
Playable character generation is a subset of AI-based image and game-asset generation. It enables the transformation of input data, such as facial images and/or with textual descriptions describing the end user necessities, into animated characters or stylized visuals resembling cartoons or anime.
This technology leverages advancements in deep learning, computer vision, and 3D modeling to produce unique and dynamic visuals that can mirror human expressions and poses. In recent years, industries have increasingly adopted anime and character generation for:
- Entertainment & Gaming
- Social Media & Virtual Reality (VR)
- Education & Training Simulations
In this project, the goal is to develop a system that generates game characters based on photorealistic images and the necessities that the end user expects in terms of textual prompts, which can be exported to game engines and VR environments seamlessly.
Methodology
Our system leverages open-source models which then trained on downstream tasks such as image-to-image translation. We have fine-tuned Stable Diffusion and Flux on a custom dataset, which serves as the underlying inference engine.
Architecture Diagram
System Components
Our system consists of three main modules:
- Data Creation: Generates datasets for downstream training.
- Generator Module: Performs real-time inference using a diffusion-based model.
- 3D Conversion: Converts 2D generated images into 3D meshes, essential for game engines and VR environments.
Current Progress
So far, we have completed the Data Creation and Generator Module stages.
Our project’s intermediate results can be found here. (under the heading Game Character Generation)
Next Steps
Our next focus is to explore various research methodologies for 3D conversion. One promising approach is the TRELLIS Model, which yields relatively good results but lacks detailed facial elements.
Expected Outcomes
-
An end-to-end system capable of generating personalized RPG characters.(a character in the fictional world of the game whose actions the player controls).
-
Seamless export to game engines and VR environments.
-
Easy customization through text prompts.
Resources & Funding
Funding will be utilized for:
- Acquiring and outsourcing 3D data
- GPU resource costs
- Model training expenses
- Software licenses (game engines/ VR environment)