LinguaLinker Icon LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

Tencent
*Equal contribution
πŸ‘¨:low πŸ‘„:high πŸ‘€:low
πŸ‘¨:low πŸ‘„:high πŸ‘€:high
πŸ‘¨:high πŸ‘„:low πŸ‘€:low
πŸ‘¨:high πŸ‘„:high πŸ‘€:high
Ref Image
Chinese
English
French

Logo LinguaLinker

Abstract

This study delves into the intricacies of synchronizing facial dynamics with multilingual audio inputs, focusing on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. Diverging from traditional parametric models for facial animation, our approach adopts a holistic diffusion-based framework that integrates audio-driven visual synthesis to enhance the synergy between auditory stimuli and visual responses. We process audio features separately and derive the corresponding control gates, which implicitly govern the enhancement of movements in lips, eyes, and head, irrespective of the portrait’s origin. The advanced audio-driven visual synthesis mechanism provides nuanced control over a diverse range of expressions and postures, allowing for a more tailored and effective portrayal of distinct personas across different languages. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the richness of motion variations achieved by our method render it a versatile tool for animating any portrait in any language.

LinguaLinker

Framework

LinguaLinker

Control Enhancement

Implicit Frequency Control

As LinguaLinker implicitly split the control signal to head, mouth and eyes, we could adjust the corresponding parameters to influence the generated portraits, separately control the motion range and motion frequency of different regions related to the character.

πŸ‘¨:low πŸ‘„:high πŸ‘€:low
πŸ‘¨:low πŸ‘„:high πŸ‘€:high
πŸ‘¨:high πŸ‘„:low πŸ‘€:low
πŸ‘¨:high πŸ‘„:high πŸ‘€:high
πŸ‘¨:low πŸ‘„:high πŸ‘€:low
πŸ‘¨:low πŸ‘„:high πŸ‘€:high
πŸ‘¨:high πŸ‘„:low πŸ‘€:low
πŸ‘¨:high πŸ‘„:high πŸ‘€:high

Generated Videos

Any portrait in any language

French
English
Chinese
Japenese
Korean
Thai

Talking with Real character & Virtual Character

Singing in Lento Rhythm & Rapid Rhythm