Deepfake Dubs: Flawless Films for the Foreign Market

Kenton Reynolds, Writer
@uxconnections

Flawless are using Deepfake technology in order to change the way that films are sold overseas.

Films and TV are almost always produced in a single language which limits the viewing audience to just a few countries. For acclaimed films that experience massive demand, they will often be wanted in countries where the language is not spoken or understood. Filmmakers are left with just a few options. The option which produces the best quality is to completely remake the film, but this is extremely costly and also loses the original cast’s performances. A slightly cheaper alternative is to dub the film, hiring voice actors to record a new soundtrack that is placed over the original film. Though this means that the viewer still sees the original cast, the sound is often clumsy and out of sink as it is clear that the actor is saying something other than what the viewer can hear; an experience that can be frustrating for the viewer. The final and cheapest option is to watch the original film with subtitles overlayed. This is the most inconvenient to the viewer as they are often unable to concentrate on the action as they are busy reading. In addition, many younger viewers are effectively rendered unable to watch the film. Flawless are providing a fourth option that is both the best for the watcher and cheap.

Flawless is an AI startup in London that uses a Neural Network Lab to edit TV & Film. They use computer algorithms to change the way that actors’ mouths move so that it appears they are speaking in a foreign language whilst a foreign dubbed soundtrack plays over the top.

Flawless are creating this effect using a Deepfake algorithm. Deepfakes have been permeating the national consciousness steadily since they were famously released via a user on Reddit in 2017. This type of algorithm was named after the user, U/deepfakes, who first released them on Reddit, though they have been researched for years academically. They use thousands of photos of an individual and thousands of photos of another. The AI collates the two different faces and learns the similarities between the two. Once the faces and their similarities have been learnt, one can be overlayed by the other on videos. For example, a video went viral which appeared to show Jimmy Fallon interviewing himself or a twin but instead had Fallon’s face placed on Paul Rudd’s body. Though this is seen as, at worst, mischievous and not in the slightest way malicious, Deepfake algorithms have developed negative connotations. This is because they have been used – often illegally – for the defamation of public figures, especially politicians, and the production of pornography. However, Flawless are changing this perception by showing one of the positive uses of this impressive technology.

One of Flawless’s promotional videos – an excerpt of which can be seen here – depicts Jack Nicholson in A Few Good Men. The clip decomposes the algorithm to five stages in order to show how the dubbing is done. The ‘Original Footage’ is inputted into the system. The system then identifies an actor’s face and takes the ‘face off’ as well as simplifying the face down from a real image to a computational model. The system then receives the new, translated soundtrack and, during the ‘performance blending’ phase, recreates the lip movements to form the words spoken in the foreign language and blends this movement into the actor’s face. Next, ‘neural rendering’ occurs which transforms the computational model back to an image which shows facial features, expression and mannerisms; the animated movements come to life here. This rendered image is then placed back onto the film in ‘face on’ and we are left with the ‘final product’.

This process, though it may appear detailed and lengthy, is fairly straightforward once the code that runs it has been created. As a result, the cost is cheap for filmmakers – especially in comparison to alternatives like the production of a remake. It also allows the move it retain the original performance of actors whilst using the new language.

Currently, the streaming market is becoming incredibly saturated in both the US and UK as people are switching to services like Netflix rather than traditional television viewing. With this success, many firms are looking to develop the same market overseas but are faced with a choice of editing their current archive of films to show a new language or buy the rights to a whole new library. Both are expensive currently, if done right, or can cause a cost to the utility of the user. Flawless will mean that this expansion will be easier at more American and British films can be translated and shown abroad for a lower cost.

It will also affect those of us in the UK or US. Increasing global connectivity over the last decade has meant we are more aware than ever of the great films industries in other countries. Parasite, A Korean film released in 2019, was incredibly popular and award-winning across the globe despite anyone unable to understand the language being forced to watch with subtitles. Flawless could mean films like Parasite are more accessible in the US and UK.

The algorithm is far from perfect just yet. One native French speaker who watched a production of Flawless that had been converted form English, said it was good but still noticeable. Though, with time it is sure to improve and leave a better quality, cheaper and viable option to spread media globally whilst retaining the actors’ performances.