Abstract: In traditional methods of RGB image guided depth image super-resolution, the reference
images are required to be high-resolution intensity images and its resolution determines the upper limit
of the depth image upsampling. Moreover, in some situations only low-resolution RGB images are
available, thus the traditional methods are unpractical. In this paper an arbitrary resolution RGB image
guided depth image super-resolution is proposed. First, we use different image super-resolution
algorithm for the input RGB image upsampling, so that a high-resolution reference RGB image can be
obtained. Then we increase the resolution of the input depth image by using the second-order total
generalized variation based method and adding edge cues from the reference image obtained in above
step. Then the final energy objective function is defined and depth image super-resolution can be
transformed into optimization problem, which can be solved by primal-dual energy minimization
scheme. Finally the high-resolution depth image is generated. This paper explores the cases previously
ignored by the relevant method and the proposed method can be applied to arbitrary resolution RGB
images. Through the relevant experiments, we found an amazing phenomenon that, by using low-resolution color image up-sampling as a guide, we can get similar to or even better results
compared with using high-resolution intensity guided image. This conclusion has some reference
significance for the research and application of related issues.

Key words: super-resolution reconstruction, depth image, second order total generalized variation, ToF
camera