Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition

Jiaxiang Tang1, Kaisiyuan Wang2, Hang Zhou2, Xiaokang Chen1, Dongliang He, Tianshu Hu2, Jingtuo Liu2, Gang Zeng1, Ziwei Liu3, Jingdong Wang2

1 Peking University   2 Baidu Inc.   3 S-Lab, Nanyang Technological University  


(Videos synthesized by our method, and audio synthesized by Speechify from text.)


While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their potential usage. In this paper, we propose an efficient NeRF-based framework that enables real-time synthesizing of talking portraits and faster convergence by leveraging the recent success of grid-based NeRF. Our key insight is to decompose the inherently high-dimensional talking portrait representation into three low-dimensional feature grids. Specifically, a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3D spatial grid and a 2D audio grid. The torso is handled with another 2D grid in a lightweight Pseudo-3D Deformable Module. Both modules focus on efficiency under the premise of good rendering quality. Extensive experiments demonstrate that our method can generate realistic and audio-lips synchronized talking portrait videos, while also being highly efficient compared to previous methods.


Our key insight is to explicitly decompose the inherently high-dimensional audio-guided portrait representation into three low-dimensional trainable feature grids:


Comparison Results

Test Set Self-driven



MakeItTalk Wav2Lip Ours
MakeItTalk Wav2Lip Ours

Video Manipulation

Our method supports various video manipulation such as eye blinks, head poses, and backgrounds.


  title={Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition},
  author={Tang, Jiaxiang and Wang, Kaisiyuan and Zhou, Hang and Chen, Xiaokang and He, Dongliang and Hu, Tianshu and Liu, Jingtuo and Zeng, Gang and Wang, Jingdong},
  journal={arXiv preprint arXiv:2211.12368},