Wednesday, July 28, 2021
No menu items!

Is Tom Hanks multi-lingual? No, it’s just AI

Bad lip-syncing in dubbing and subtitles have always put off audiences wherever they are in the world, box office takings of foreign films show. AI may be about to change all that. Francesca Lynagh reports.

Is Tom Hanks multi-lingual?

No, he’s just getting a little help from AI.

Startup Flawless AI has been working on making dubbing in films less… distracting.

The company studies in intricate detail all the mouth movements an actor makes, and swaps them out according to the dubbed words in different languages.

The end result makes Jack Nicholson look like he’s fluent in French.

Scott Mann is a film director and co-founder of Flawless AI.

“Look, personally, I hate dubbing, because you have to change so many things to try and catch sync so you’re changing words, words that filmmakers and performers we’ve kind of all thought about so deeply but they’re just kind of thrown out to find a different word that kind of fits sync but it never really does. Like whenever we watch a dub, we always know it’s not in sync and what that leads to is a non-immersive experience and so it pulls you away from feeling films and feeling content. And that’s damaging and it really lessens the experience and it puts people off wanting to do it, you know.”

The idea began when Mann saw how dubbing affected the narrative cohesion of his 2015 film “Heist”, starring Robert De Niro.

After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.

“Essentially it’s studying in intricate detail all the nuance of human pronunciation and all the visemes that are made during mouth movement and it’s able to essentially take, say, an ‘ooh’ sound from De Niro 20 minutes earlier and place that in a different moment in the film and it kind of measures at the same and blends it so that the performance is the same but it’s a different mouth movement.”

Flawless AI is now working with producers and studios to integrate the technology into post-production.

Mann believes that in time, the tool will help eradicate foreign language categories at award shows, and create a more inclusive film industry with a real global stage.

He said the first films using this tool are around a year away.

“I know that the first few things that come out, people will probably be looking out for it in particular and will be studying it. We’ll all do it, I’m sure and it’s interesting like that but look. We’re all about doing this at scale for everyone and I think when it’s a common thing we’re not going to think about it anymore and we’ll just enjoy the movies.”

Latest News

Fort Bend County launches new web address and mobile app

(FORT BEND, TX) – Fort Bend County government has a new home on the Web. Today, as...

Biles puts mental health first after exit, unsure of upcoming events

Simone Biles' bid for a record six gold medals at the Tokyo Games ended abruptly on Tuesday (July 27) when she dropped...

Apple’s iPhone expected to drive sales, but App Store faces regulatory risk

Strong sales of 5G iPhones and services such as the App Store and Apple Music are expected to drive growth at Apple...

U.S. CDC recommends vaccinated Americans wear masks indoors in many cases

Americans fully vaccinated against COVID-19 should go back to wearing masks in indoor public places in regions where the coronavirus and especially...

Four pedestrians including 2 children struck by vehicle in Hempstead

HOUSTON – At least four people, including two children, were injured after a vehicle struck them in Hempstead Monday night.

More Articles Like This