Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion

Abstract

In this paper, we propose our Correlation For Completion Network (CFCNet), an end-to-end deep model to do the sparse depth completion task with RGB information. We first propose a 2D deep canonical correlation analysis as network constraints to ensure encoders of RGB and depth capture the most similar semantics. We then transform the RGB features to the depth domain. The complementary RGB information is used to complete the missing depth information. We conduct extensive experiments on both outdoor and indoor scene datasets. For outdoor scenes, KITTI and Cityscape are used, which captured the depth information with LiDARs and stereo cameras respectively. For indoor scenes, we use NYUv2 with stereo/ORB feature sparsifiers and SLAM RGBD datasets. Experiments demonstrate our CFCNet outperforms the state-of-the-art methods using these datasets. Our best results improve the percentage of accurate estimations from 13.03 to 58.89 (+394%) compared with the state-of-the-art method on the SLAM RGBD dataset.