Facebook AI Creates 3D Avatars That Look Exactly Like You To Play In Video Games

In the past many months, we have been feeling upset with Facebook in several aspects, in fact, we have moved from being upset to feeling utterly normal with every mistake and word-only promises from the social giant. However, the latest news about AI may make you smile a little, hopefully.

In particular, its AI research team has developed a way in which they can turn a real human into a playable game character through analyses of videos recording them in specific motions. With this innovation, we can actually play in our video games.

In fact, the idea of using real people footage to make video games has been around for decades. Dated back the 90s, Time Traveler was the first holographic game to dig into a gameplay experience by displaying pre-recorded videos corresponding to players’ choices. Moreover, in Mortal Kombat, its first version was made by recording costumed characters on a sound stage, but then the footage was turned into animated sprites to make sure the game went smoothly.

In today video games, most characters are fully 3D models, and even though players can customize a lot of things to make their models unique to themselves, the movements are still limited to stock animation.

The newly published research by the AI Research division at Facebook could be the game changer. Accordingly, the team trained two distinct neural networks using footage of 5 to 8 minutes long, showing a person doing a particular action, playing tennis for example. The first is Pose2Pose for analyzing the footage and extracting the person making those motions. The second is Pose2Frame for transferring all of the person’s elements including reflections and shadows. Then, it is overlaid on a new background setting that could be a rendered video game locale.

Though the results are not as smooth and fluid as the 3D models in present games, they are definitely controllable. The results will certainly improve when the research evolves, but it can get even better with a hybrid approach. The AI is able to extract someone’s characteristics from a video, and automatically apply those to create a 3D character, which means players no longer have to custom every detail by themselves.

The applications of this research are beyond video games, we can use it to create believable avatars of ourselves more easily, and it may take just a few minutes to do that.