Abstract: To address the problems of the poor filling effect of the video image defect in video image stabilization, which seriously affects the visual effect and causes the black edge filling of the video after image stabilization processing, an image repair method was proposed based on time series network prediction and pyramid fusion. First, the pre-cutting mechanism was employed to adaptively determine whether the current frame needed to be repaired. Then all frames up to the current moment were sent to the model combining convolutional neural networks (CNN) and gated recurrent unit (GRU) to predict the part to be filled. Next, the improved weighted optimal stitching was used for stitching and image fusion reconstruction in the Gaussian Laplace pyramid. Finally, the size was cut after the completion of reconstruction. The experimental results show that the average peak signal to noise ratio (PSNR) of the method was 2–5 dB higher than that of the compared algorithm, and that the average structural similarity (SSIM) was improved by about 2%–7%. In addition, the video defect repaired by this method exhibits a natural filling effect and a relatively stable visual effect. Even in the cases of large black areas, the repair performance remains stable, which can be applied to a variety of camera platforms and different scenarios. 

Key words: video stabilization, video inpainting, time series network, pyramid fusion, optimal seam