Abstract: Aiming at the problems of poor real-time performance and low accuracy faced by the current virtual and
real occlusion processing technology based on depth information, a real-time virtual and real occlusion processing
algorithm was proposed based on local area depth estimation and patch similarity-based fusion of noise points. The
algorithm took the real scene video sequence as the input. Firstly, the depth information of sparse key points was
estimated using the local area depth estimation algorithm through sparse reconstruction, and the sparse depth was
imposed on the target area to limit the depth propagation to the surrounding pixels, so as to quickly restore the relative
depth map of the target area. Then, the noise point voting fusion algorithm employed depth comparison to determine the front and back position relationship of the virtual and real objects, and voted and merged the pixels in the area
based on the patch similarity and voting decision method. Finally, the fusion effect with real occlusion relationship
was output. The experimental results show that the proposed algorithm can not only improve the real-time
performance of virtual and real occlusion, but also obtain a good fusion effect under different spatial relationships
between real and virtual scenes.

Key words: occlusion, depth estimation, regional constraints, voting decision, patch similarity