Your favorite TV characters now as AI ghosts!
Starting with Sheldon Cooper from the Big Bang Theory!
Paper: Hi Sheldon! Creating Deep Personalized Characters from TV Shows
Researchers at Tsinghua University (among other contributors) want to make AI ghosts of their favorite TV show characters.
Mainly, They wanted to use AI to create multi modal Deep Personalized Character Creation (DPCC) copy of the appearance and personality of the characters.
More specifically, given an input (text, audio, video), the goal of DPCC is to generate a multi-modality (text, audio, video) response, which should match the personality of a specific character such as Sheldon, to a high quality level.
First they collect character-centric conversations, from raw videos, audio, transcripts, and sub-titles from a TV show. They choose The Big Bang since it has large number of episodes with the characters include clean facial expressions, vocals and texts.
They trained DialogGPT, Text-To-Speech and Pose-Controllable Talking Face Generation Model(PC-AVS) to combine text, vocals and video of characters
Finally, they did subjective and objective evaluation for 5 DeepCharacters: DeepSheldon, DeepPenny,DeepLeonard, DeepHoward and DeepRaj
The created DeepCharacters is capable of generating relatively high-quality response, and consistent personality to the original character. They benchmark well on objective evaluations of fluency and consistency and also are comparable in subjective evaluations.
So Essentially,
"AI can do a great job at mimicking TV characters"