欢迎访问《图学学报》 分享到：
图学学报

图学学报 ›› 2021, Vol. 42 ›› Issue (1): 65-70.DOI: 10.11996/JG.j.2095-302X.2021010065

• 图像处理与计算机视觉 • 上一篇    下一篇

结合时序网络和金字塔融合的稳像修复方法 

  

  1. 1. 江苏大学计算机科学与通信工程学院，江苏 镇江 212013；  2. 江苏大学网络空间安全研究院，江苏 镇江 212013； 3. 江苏省大数据泛在感知与智能农业应用工程研究中心，江苏 镇江 212013；  4. 中国电子科学研究院社会安全风险感知与防控大数据应用国家工程实验室，北京 100041；  5. 新疆联海创智信息科技有限公司，新疆 乌鲁木齐 830001)
  • 出版日期:2021-02-28 发布日期:2021-02-01
  • 基金资助:
    国家自然科学基金项目(61972183，61672268)；社会安全风险感知与防控大数据应用国家工程实验室主任基金项目(201807) 

Image stabilization repair method combining time series network and pyramid fusion 

  1. 1. School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang Jiangsu 212013, China;  2. Cyber Space Security Academy of Jiangsu University, Zhenjiang Jiangsu 212013, China;  3. Jiangsu Province Big Data Ubiquitous Perception and Intelligent Agricultural Application Engineering Research Center, Zhenjiang Jiangsu 212013, China;  4. National Engineering Laboratory for Public Security Risk Perception and Control by Big Data, China Academy of Electronic Sciences, Beijing 100041, China;  5. Xinjiang Lianhaichuangzhi Information Technology Co., Ltd., Urumqi Xinjiang 830001, China
  • Online:2021-02-28 Published:2021-02-01
  • Supported by:
    National Natural Science Foundation of China (61972183, 61672268); National Engineering Laboratory Director Foundation of Big Data Application for Social Security Risk Perception and Prevention (201807)

PDF (PC)

68

摘要： 针对视频稳像领域内视频图像缺损填充效果不佳，严重影响视觉效果，且导致稳像处理后的视 频不稳的黑边填充问题，提出了一种基于时序网络预测和金字塔融合的图像修复方法。首先结合预裁剪机制自 适应判断当前帧是否需修复；然后将截止至当前时刻的所有帧送入卷积神经网络(CNN)和门控循环单元(GRU) 的模型进行待填充部分的预测；随后采用改进的加权最佳缝合线进行拼接并在高斯拉普拉斯金字塔中进行图像 融合重构；最终在重构完成后裁剪尺寸。实验结果表明，该方法平均峰值信噪比(PSNR)相较于对比算法提高了 2~5 dB，平均结构相似度(SSIM)较对比算法提升了约 2%~7%。该方法修复后的视频缺损填充自然，视觉效果 较为稳定，即使在黑边面积较大时也有良好的修复效果，可用于多种摄像平台及不同场景下。

关键词: 视频稳像, 视频图像修复, 时序网络, 金字塔融合, 最佳缝合线

Abstract: To address the problems of the poor filling effect of the video image defect in video image stabilization, which seriously affects the visual effect and causes the black edge filling of the video after image stabilization processing, an image repair method was proposed based on time series network prediction and pyramid fusion. First, the pre-cutting mechanism was employed to adaptively determine whether the current frame needed to be repaired. Then all frames up to the current moment were sent to the model combining convolutional neural networks (CNN) and gated recurrent unit (GRU) to predict the part to be filled. Next, the improved weighted optimal stitching was used for stitching and image fusion reconstruction in the Gaussian Laplace pyramid. Finally, the size was cut after the completion of reconstruction. The experimental results show that the average peak signal to noise ratio (PSNR) of the method was 2–5 dB higher than that of the compared algorithm, and that the average structural similarity (SSIM) was improved by about 2%–7%. In addition, the video defect repaired by this method exhibits a natural filling effect and a relatively stable visual effect. Even in the cases of large black areas, the repair performance remains stable, which can be applied to a variety of camera platforms and different scenarios. 

Key words: video stabilization, video inpainting, time series network, pyramid fusion, optimal seam

中图分类号: