Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks.However,current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template,hindering use in real applications.To address these problems,we propose a deep shape reconstruction and texture completion network,SRTC-Net,which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image.In SRTC-Net,we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes.The SRTC-Net pipeline has three stages.The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model,and transfers the input 2D image to a U-V texture map.Then we complete the invisible and occluded areas in the U-V texture map using an inpainting network.To get the 3D facial geometries,we predict coarse shape(U-V position maps)from the segmented face from the correspondence network using a shape network,and then refine the 3D coarse shape by regressing the U-V displacement map from the completed U-V texture map in a pixel-to-pixel way.We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks,using both in-the-lab datasets(MICC,MultiPIE)and in-the-wild datasets(CFP).The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture;they outperform or are comparable to the state-of-the-art.

Xiaoxing Zeng;Zhelun Wu;Xiaojiang Peng;Yu Qiao

Shenzhen Institute of Advanced Technology,Chinese Academy of Sciences,Shenzhen,China;University of Chinese Academy of Sciences,Beijing,China

计算可视媒体（英文）

[1]Xiaoxing Zeng;Zhelun Wu;Xiaojiang Peng;Yu Qiao-.Joint 3D facial shape reconstruction and texture completion from a single image)[J].计算可视媒体（英文）,2022(02):239-256

SRTC,regressing,frontalization,MICC,MultiPIE

Joint,facial,reconstruction,texture,completion,from,single,image,Recent,years,have,witnessed,significant,progress,face,using,deep,convolutional,neural,networks,However,current,methods,often,improperly,self,occluded,regions,lead,inaccurate,correspondences,between,2D,input,template,hindering,use,real,applications,To,address,these,problems,propose,Net,which,jointly,reconstructs,geometry,completes,In,leverage,geometric,cues,completed,detailed,structures,shapes,pipeline,has,three,stages,first,introduces,identify,pixel,wise,model,transfers,Then,invisible,areas,inpainting,get,geometries,predict,coarse,position,maps,segmented,then,refine,by,displacement,way,We,examine,our,tasks,well,invariant,recognition,both,lab,datasets,wild,CFP,qualitative,quantitative,results,demonstrate,effectiveness,inferring,they,outperform,comparable,state,art

0.469983