Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition


Jiaxiang Tang1, Kaisiyuan Wang2, Hang Zhou2, Xiaokang Chen1, Dongliang He, Tianshu Hu2, Jingtuo Liu2, Gang Zeng1, Ziwei Liu3, Jingdong Wang2

1 Peking University   2 Baidu Inc.   3 S-Lab, Nanyang Technological University  

Demonstration


(Videos synthesized by our method, and audio synthesized by Speechify from text.)

Abstract


While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their potential usage. In this paper, we propose an efficient NeRF-based framework that enables real-time synthesizing of talking portraits and faster convergence by leveraging the recent success of grid-based NeRF. Our key insight is to decompose the inherently high-dimensional talking portrait representation into three low-dimensional feature grids. Specifically, a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3D spatial grid and a 2D audio grid. The torso is handled with another 2D grid in a lightweight Pseudo-3D Deformable Module. Both modules focus on efficiency under the premise of good rendering quality. Extensive experiments demonstrate that our method can generate realistic and audio-lips synchronized talking portrait videos, while also being highly efficient compared to previous methods.


Method


Our key insight is to explicitly decompose the inherently high-dimensional audio-guided portrait representation into three low-dimensional trainable feature grids:

teaser

Comparison Results


Test Set Self-driven

GT AD-NeRF Ours

Cross-driven

NVP LSP AD-NeRF Ours
MakeItTalk Wav2Lip Ours
MakeItTalk Wav2Lip Ours

Video Manipulation


Our method supports various video manipulation such as eye blinks, head poses, and backgrounds.


Citation


@article{tang2022radnerf,
  title={Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition},
  author={Tang, Jiaxiang and Wang, Kaisiyuan and Zhou, Hang and Chen, Xiaokang and He, Dongliang and Hu, Tianshu and Liu, Jingtuo and Zeng, Gang and Wang, Jingdong},
  journal={arXiv preprint arXiv:2211.12368},
  year={2022}
}