基于张量低秩分解和非下采样剪切波变换的视频图像去雪方法

doi:10.11996/JG.j.2095-302X.2023050947

图学学报 ›› 2023, Vol. 44 ›› Issue (5): 947-954.DOI: 10.11996/JG.j.2095-302X.2023050947

• 图像处理与计算机视觉 • 上一篇下一篇

基于张量低秩分解和非下采样剪切波变换的视频图像去雪方法

张云鹏¹(), 周浦城¹^,²(), 薛模根¹^,²

1.偏振光成像探测技术安徽省重点实验室，安徽合肥 230031
2.陆军炮兵防空兵学院信息工程系，安徽合肥 230031

收稿日期:2023-03-09 接受日期:2023-05-26 出版日期:2023-10-31 发布日期:2023-10-31
通讯作者: 周浦城(1977-)，男，教授，博士。主要研究方向为图形图像处理、计算机视觉等。E-mail：zhoupc@hit.edu.cn
作者简介:张云鹏(1993-)，男，硕士研究生。主要研究方向为数字图像处理。E-mail：1791973191@qq.com
基金资助:
国家自然科学基金项目(61379105);安徽省自然科学基金项目(1908085MF208)

Snow removal in video based on low-rank tensor decomposition and non-subsampled shearlet transform

ZHANG Yun-peng¹(), ZHOU Pu-cheng¹^,²(), XUE Mo-gen¹^,²

1. Anhui Province Key Laboratory of Polarization Imaging Detection Technology, Hefei Anhui 230031, China
2. Department of Information Engineering, PLA Army Academy of Artillery and Air Defense, Hefei Anhui 230031, China

Received:2023-03-09 Accepted:2023-05-26 Online:2023-10-31 Published:2023-10-31
Contact: ZHOU Pu-cheng (1977-), professor, Ph.D. His main research interests cover graphic image processing, computer vision, etc. E-mail：zhoupc@hit.edu.cn
About author:ZHANG Yun-peng (1993-), master student. His main research interest covers digital image processing. E-mail：1791973191@qq.com
Supported by:
National Natural Science Foundation of China(61379105);Anhui Provincial Natural Science Foundation(1908085MF208)

摘要/Abstract

摘要：

在雪天条件下，雪花会对视频监控系统造成遮挡，使部分重要景物信息无法被捕获，严重降低获取的视频图像质量，对后续目标检测与识别等高级图像处理造成强烈干扰。现有视频图像去雪方法普遍存在去雪效果不稳定，运算耗时长等缺陷。针对该问题，首先利用张量能充分挖掘视频图像中蕴含的空间位置信息的优势，通过张量低秩分解模型与三维全变分正则化相结合，将受到雪花污染的监控视频分解为静态背景层和运动前景层；然后，采用基于非下采样剪切波变换和数学形态学滤波方法，将运动前景层进一步分解为运动物体层和雪层；最后，将静态背景层和运动物体层进行重构，得到最终去雪后的视频图像。实验结果表明，该方法能够有效去除视频图像中的雪花干扰，清晰保留景物边缘信息，在处理效果和运算效率上均优于现有先进算法。

关键词: 张量低秩分解, 非下采样剪切波变换, 视频去雪, 数学形态学滤波

Abstract:

Under snowy conditions, snowflakes can obstruct video surveillance systems, preventing the capture of important scenery information and drastically reducing the quality of acquired video images. This interference can also strongly affect advanced image processing techniques such as subsequent target detection and recognition. The existing methods for removing snow from video images commonly suffer from drawbacks such as unstable snow removal performance and long computational time. To address this issue, firstly, the advantages of tensors in fully capturing spatial location information within video images were leveraged. By combining a low-rank tensor decomposition model with three-dimensional total variation regularization, the snow-contaminated surveillance video was decomposed into a static background layer and a moving foreground layer. Then, based on the non-subsampled shearlet transform and mathematical morphology filtering methods, the moving foreground layer was further decomposed into a moving object layer and a snow layer. Finally, the static background layer and moving object layer were reconstructed to obtain snow-free video images. The experimental results demonstrated the effectiveness of this approach in removing snowflake interference from video images while clearly retaining scene edge information. Moreover, the proposed method outperforms existing state-of-the-art algorithms in terms of processing efficacy and operational efficiency.

Key words: low-rank tensor decomposition, non-subsampled shearlet transform, snow removal in video, mathematical morphology filtering

中图分类号:

TP391

张云鹏, 周浦城, 薛模根. 基于张量低秩分解和非下采样剪切波变换的视频图像去雪方法[J]. 图学学报, 2023, 44(5): 947-954.

ZHANG Yun-peng, ZHOU Pu-cheng, XUE Mo-gen. Snow removal in video based on low-rank tensor decomposition and non-subsampled shearlet transform[J]. Journal of Graphics, 2023, 44(5): 947-954.

图/表 8

图1 算法处理流程

Fig. 1 Overview of the proposed method

图2 合成雪视频去雪效果对比((a)合成雪视频图像；(b) MAWRNe；(c) HDCWNet；(d) JSTASR；(e) DesnowNet；(f) AirNet；(g) LRMC；(h) MSCSC；(i) OTMSCSC；(j) Ours)

Fig. 2 Comparison of snow removal results of synthetic snow videos ((a) Video with synthetic snow.; (b) MAWRNe; (c) HDCWNet; (d) JSTASR; (e) DesnowNet; (f) AirNet; (g) LRMC; (h) MSCSC; (i) OTMSCSC; (j) Ours)

表1 不同算法去雪效果评价结果

Table 1 Evaluation results of different snow removal methods

算法	PSNR(dB)	SSIM	FSIM	RMSE
合成雪天视频图像	33.467 0	0.969 3	0.952 8	5.409 9
MAWRNet	35.741 8	0.976 7	0.981 0	4.163 4
HDCWNet	20.891 7	0.897 4	0.939 4	23.012 0
JSTASR	23.919 2	0.872 7	0.930 3	16.239 7
DesnowNet	30.998 0	0.963 7	0.955 6	7.188 5
AirNet	32.644 7	0.968 7	0.954 7	5.974 1
LRMC	37.412 1	0.976 4	0.992 1	3.435 1
MSCSC	36.367 6	0.973 9	0.988 3	3.874 0
OTMSCSC	35.857 6	0.976 8	0.989 9	4.108 7
本文算法	38.098 8	0.979 6	0.995 6	3.173 9

图3 静态背景场景去雪效果对比((a)雪天视频图像；(b) MAWRNet；(c) HDCWNet；(d) JSTASR；(e) DesnowNet；(f) AirNet；(g) LRMC；(h) MSCSC；(i) OTMSCSC；(j)本文算法)

Fig. 3 Comparison of snow removal effect for stationary background scene ((a) Video with real snow; (b) MAWRNet; (c) HDCWNet; (d) JSTASR; (e) DesnowNet; (f) AirNet; (g) LRMC; (h) MSCSC; (i) OTMSCSC; (j) Ours)

图4 动态背景场景去雪效果对比((a)雪天视频图像；(b) MAWRNet；(c) HDCWNet；(d) JSTASR；(e) DesnowNet；(f) AirNet；(g) LRMC；(h) MSCSC；(i) OTMSCSC；(j)本文算法)

Fig. 4 Comparison of snow removal effect for dynamic background scene ((a) Video with real snow; (b) MAWRNet; (c) HDCWNet; (d) JSTASR; (e) DesnowNet; (f) AirNet; (g) LRMC; (h) MSCSC; (i) OTMSCSC; (j) Ours)

图5 存在运动物体场景去雪效果对比((a)雪天视频图像；(b) MAWRNet；(c) HDCWNet；(d) JSTASR；(e) DesnowNet；(f) AirNet；(g) LRMC；(h) MSCSC；(i) OTMSCSC；(j)本文算法)

Fig. 5 Comparison of snow removal effect for moving object scene ((a) Video with real snow; (b) MAWRNet; (c) HDCWNet; (d) JSTASR; (e) DesnowNet; (f) AirNet; (g) LRMC; (h) MSCSC; (i) OTMSCSC; (j) Ours)

表2 不同视频去雪算法的定量评价结果

Table 2 Comparison of evaluation index for different snow removal algorithms

算法	图3			图4			图5
算法	PIQE↓	NIQE↓	BRISQUE↓	PIQE↓	NIQE↓	BRISQUE↓	PIQE↓	NIQE↓	BRISQUE↓
原始图像	57.096 9	3.093 0	30.038 8	42.008 5	6.336 4	27.307 5	46.722 9	4.362 3	26.678 2
MAWRNet	51.574 4	3.151 4	30.131 8	33.379 0	4.028 9	12.518 5	24.237 5	2.982 5	11.384 1
HDCWNet	55.649 1	3.764 4	26.802 2	29.474 1	5.787 3	22.922 0	40.312 8	4.639 1	19.357 9
JSTASR	55.404 1	3.482 6	28.166 8	28.838 8	6.059 0	13.251 2	39.977 3	4.500 5	22.012 9
DesnowNet	58.487 0	3.265 2	22.623 1	39.875 0	3.988 3	13.655 9	42.540 9	4.501 7	9.358 0
AirNet	54.280 1	3.113 9	24.102 5	37.785 9	3.849 8	18.953 8	35.771 3	4.244 9	8.865 1
LRMC	58.719 4	3.708 1	23.952 0	38.958 1	3.142 8	26.065 9	36.664 0	4.312 6	8.066 7
MSCSC	56.851 6	3.124 7	29.685 1	42.400 0	2.981 6	26.015 1	37.463 9	4.040 3	10.555 4
OTMSCSC	56.508 2	3.103 4	29.086 4	38.994 3	4.140 9	26.905 0	39.287 0	4.407 5	19.684 8
本文算法	51.884 1	3.026 2	22.331 5	27.586 8	2.811 7	12.500 7	22.249 6	2.955 8	8.572 2

表3 算法耗时对比(s)

Table 3 Running time comparisons (s)

输入	LRMC	MSCSC	OTMSCSC	本文算法
图3(a)	3046	2184	1317	594
图4(a)	327	210	81	112
图5(a)	792	423	312	289

参考文献 23

[1]	周浦城, 张杰, 薛模根, 等. 基于卷积分析稀疏表示和相位一致性的低照度图像增强[J]. 电子学报, 2020, 48(1): 180-188. DOI
	ZHOU P C, ZHANG J, XUE M G, et al. Low-light image enhancement based on convolutional analysis sparse representation and phase congruency[J]. Acta Electronica Sinica, 2020, 48(1): 180-188. (in Chinese)
[2]	张学锋, 李金晶. 基于双注意力残差循环单幅图像去雨集成网络[J]. 软件学报, 2021, 32(10): 3283-3292.
	ZHANG X F, LI J J. Single image de-raining using a recurrent dual-attention-residual ensemble network[J]. Journal of Software, 2021, 32(10): 3283-3292. (in Chinese)
[3]	贾童瑶, 卓力, 李嘉锋, 等. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245. DOI
	JIA T Y, ZHUO L, LI J F, et al. Research advances on deep learning based single image dehazing[J]. Acta Electronica Sinica, 2023, 51(1): 231-245. (in Chinese)
[4]	YANG T W, NSABIMANA V, WANG B F, et al. Snow fluff detection and removal from video images[C]// Ihe 43rd Annual Conference of the IEEE Industrial Electronics Society. New York: IEEE Press, 2017: 3840-3844.
[5]	LI Y J, WU R, JIA Z H, et al. Video desnowing and deraining via saliency and dual adaptive spatiotemporal filtering[J]. Sensors, 2021, 21(22): 7610: 1-7610:18. DOI URL
[6]	KIM J H, SIM J Y, KIM C S. Video deraining and desnowing using temporal correlation and low-rank matrix completion[J]. IEEE Transactions on Image Processing, 2015, 24(9): 2658-2670. DOI URL
[7]	LI M H, XIE Q, ZHAO Q, et al. Video rain streak removal by multiscale convolutional sparse coding[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 6644-6653.
[8]	LI M H, CAO X Y, ZHAO Q, et al. Online rain/snow removal from surveillance videos[J]. IEEE Transactions on Image Processing, 2021, 30: 2029-2044. DOI PMID
[9]	CHEN W T, HUANG Z K, TSAI C C, et al. Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: toward a unified model[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 17632-17641.
[10]	CHEN W T, FANG H Y, HSIEH C L, et al. ALL snow removed: single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 4176-4185.
[11]	CHEN W T, FANG H Y, DING J J, et al. JSTASR: joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 754-770.
[12]	LIU Y F, JAW D W, HUANG S C, et al. DesnowNet: context-aware deep network for snow removal[J]. IEEE Transactions on Image Processing, 2018, 27(6): 3064-3073. DOI URL
[13]	LI B Y, LIU X, HU P, et al. All-In-one image restoration for unknown corruption[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 17431-17441.
[14]	EASLEY G, LABATE D, LIM W Q. Sparse directional image representations using the discrete shearlet transform[J]. Applied and Computational Harmonic Analysis, 2008, 25(1): 25-46. DOI URL
[15]	ZHANG X Y, YUAN X, CARIN L. Nonlocal low-rank tensor factor analysis for image restoration[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 8232-8241.
[16]	XU H Y, JIANG G Y, YU M, et al. Tensor product and tensor-singular value decomposition based multi-exposure fusion of images[J]. IEEE Transactions on Multimedia, 2022, 24: 3738-3753. DOI URL
[17]	TOM A J, GEORGE S N. A three-way optimization technique for noise robust moving object detection using tensor low-rank approximation, l1/2, and TTV regularizations[J]. IEEE Transactions on Cybernetics, 2021, 51(2): 1004-1014. DOI URL
[18]	DUAN J M, JIA X, BARTLETT J, et al. Arbitrary order total variation for deformable image registration[J]. Pattern Recognition, 2023, 137: 109318. DOI URL
[19]	KOLDA T G, BADER B W. Tensor decompositions and applications[J]. SIAM Review, 2009, 51(3): 455-500. DOI URL
[20]	WANG Y, PENG J J, ZHAO Q, et al. Hyperspectral image restoration via total variation regularized low-rank tensor decomposition[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(4): 1227-1243. DOI URL
[21]	BIOUCAS-DIAS J M, NASCIMENTO J M P. Hyperspectral subspace identification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(8): 2435-2445. DOI URL
[22]	JIANG N F, LUO J W, LIN J H, et al. Lightweight semi-supervised network for single image rain removal[J]. Pattern Recognition, 2023, 137: 109277. DOI URL
[23]	CHEN Z Y, WANG Y C, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7176-7185.

基于张量低秩分解和非下采样剪切波变换的视频图像去雪方法

Snow removal in video based on low-rank tensor decomposition and non-subsampled shearlet transform

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 23

相关文章 2

编辑推荐

Metrics

本文评价

[1]	盛佳佳1,2，杨学志 1,2，董张玉 1,2，焦玮 1,2. 基于NSST-IHS 变换稀疏表示的SAR 与可见光图像融合[J]. 图学学报, 2018, 39(2): 201-208.
[2]	冯裴裴，杨风暴，卫红，李大威，梁若飞. 基于正态DS 证据理论的机载LIDAR 数据地物分类方法[J]. 图学学报, 2015, 36(6): 926-930.