Depth information can benefit various computer vision tasks on both images and videos.However,depth maps may suffer from invalid values in many pixels,and also large holes.To improve such data,we propose a joint self-supervised and reference-guided learning approach for depth inpainting.For the self-supervised learning strategy,we introduce an improved spatial convolutional sparse coding module in which total variation regularization is employed to enhance the structural information while preserving edge information.This module alternately learns a convolutional dictionary and sparse coding from a corrupted depth map.Then,both the learned convolutional dictionary and sparse coding are convolved to yield an initial depth map,which is effectively smoothed using local contextual information.The reference-guided learning part is inspired by the fact that adjacent pixels with close colors in the RGB image tend to have similar depth values.We thus construct a hierarchical joint bilateral filter module using the corresponding color image to fill in large holes.In summary,our approach integrates a convolutional sparse coding module to preserve local contextual information and a hierarchical joint bilateral filter module for filling using specific adjacent information.Experimental results show that the proposed approach works well for both invalid value restoration and large hole inpainting.

Heng Wu;Kui Fu;Yifan Zhao;Haokun Song;Jia Li

State Key Laboratory of Virtual Reality Technology and Systems,School of Computer Science and Engineering,Beihang University,Beijing 100191,China

计算可视媒体（英文）

[1]Heng Wu;Kui Fu;Yifan Zhao;Haokun Song;Jia Li-.Joint self-supervised and reference-guided learning for depth inpainting)[J].计算可视媒体（英文）,2022(04):597-612

convolved

Joint,self,supervised,reference,guided,learning,depth,inpainting,Depth,information,can,benefit,various,computer,vision,tasks,both,images,videos,However,maps,may,suffer,from,invalid,values,many,pixels,also,large,holes,To,such,data,joint,approach,For,strategy,introduce,improved,spatial,convolutional,sparse,coding,module,which,total,variation,regularization,employed,enhance,structural,while,preserving,edge,This,alternately,learns,dictionary,corrupted,Then,learned,are,yield,initial,effectively,smoothed,using,local,contextual,part,inspired,by,fact,that,adjacent,close,colors,RGB,tend,have,similar,We,thus,construct,hierarchical,bilateral,filter,corresponding,In,summary,our,integrates,preserve,filling,specific,Experimental,results,show,proposed,works,well,restoration

0.533674