A generative network based on non-local information for atmospheric polarization modelling

doi:10.11996/JG.j.2095-302X.2023030551

Abstract

Abstract:

As a stable natural attribute, the atmospheric polarization mode is widely used in various fields such as navigation and detection because it contains the ∞ shape feature and meridian feature with directional information. However, obtaining atmospheric polarization information in real weather conditions is a challenging task due to the limitations imposed by dynamic cloud interference. This limitation causes the distribution law of atmospheric polarization information to be destroyed and, in turn, leads to the loss of some information. To address this issue, we proposed an atmospheric polarization mode generation method based on non-local information. Then, a non-local information inpainting block was designed for two-stage repair. In the first stage, the non-local spatial continuity information of the atmospheric polarization mode was mined to enhance the global structure between feature information and realize spatial information repair. In the second stage, the feature mapping relationship was established between atmospheric polarization information at different times, and the time continuity of the non-local atmospheric polarization information distribution was employed to repair feature information of superimposed noise regions in the time dimension. The experimental results on the Temporal Polarization 1072 polarization dataset qualitatively and quantitatively demonstrated the efficacy of this method in effectively removing cloud interference noise in the atmospheric polarization mode and repairing polarization information of missing areas, and higher structural and semantic consistency of the generated results.

Key words: atmospheric polarization mode, cloud interference, non-local information, spatial dimension, time dimension

CLC Number:

TP391

YAN Yuan, GAO Xin-jian, GAO Jun, WANG Xin, CHENG Qian. A generative network based on non-local information for atmospheric polarization modelling[J]. Journal of Graphics, 2023, 44(3): 551-559.

Figures/Tables 10

Fig. 1 The comparison chart of atmospheric polarization information under different weather conditions ((a) Collected data; (b) Processed data)

Fig. 2 The overall network framework

Table 1 Encoder composition

Kernel	Stride	Dilation	Outputs
5×5	1×1	1	64
3×3	2×2	1	128
3×3	2×2	1	256
3×3	1×1	1	256
3×3	1×1	2	256
3×3	1×1	4	256
3×3	1×1	8	256

Fig. 3 The diagram of non-local information inpainting block

Fig. 4 The visual schematic of time information repair process

Fig. 5 The diagram of the reconstructed results in different weather ((a) The reconstruction results of sunny weather data; (b) The reconstruction results of cloudy weather data)

Fig. 6 The reconstruction results under different area cloud interference conditions ((a) Original information; (b) Input information; (c) Reconstructed information; (d) Difference image)

Fig. 7 The schematic diagram of the results of various model reconstructions in cloud weather ((a) Original information; (b) Input information; (c) CPN; (d) CRA; (e) CDS-VQ-VAE; (f) GNAPM; (g) Ours)

Table 2 The comparison of PSNR and SSIM of each method reconstruction result

指标	Method	Proportion of cloud area (%)
指标	Method	Under 20	20~40	41~60	61~80	Over 80
PSNR	CPN	25.21	21.34	16.86	11.55	11.07
	CRA	24.11	20.26	15.57	12.24	11.76
	CDS-VQ-VAE	24.49	21.03	16.02	13.21	10.95
	GNAPM	26.00	22.61	18.46	15.35	12.98
	Ours	25.36	23.47	20.32	18.21	16.53
SSIM	CPN	0.970	0.864	0.804	0.699	0.638
	CRA	0.946	0.822	0.773	0.704	0.674
	CDS-VQ-VAE	0.964	0.862	0.795	0.724	0.631
	GNAPM	0.973	0.876	0.817	0.731	0.703
	Ours	0.974	0.899	0.861	0.823	0.802

Table 3 The result of ablation experiments

云层干扰面积	无时间信息修复层的生成网络		有时间信息修复层的生成网络
云层干扰面积	PSNR	SSIM	PSNR	SSIM
(0~20%)	24.12	0.965	25.34⬆5%	0.972⬆1%
(20%~40%)	21.86	0.858	23.61⬆8%	0.903⬆5%
(40%~60%)	15.43	0.769	21.02⬆36%	0.884⬆15%
(60%~80%)	13.46	0.701	18.24⬆36%	0.841⬆19%
(80%~100%)	12.35	0.686	16.91⬆37%	0.789⬆15%

References 30

[1]	CHOI H, LEE K M, SEO J, et al. The influence of atmospheric composition on polarization in the GEMS spectral region[J]. Asia-Pacific Journal of Atmospheric Sciences, 2021, 57(3): 587-603. DOI
[2]	FOSTER J J, TEMPLE S E, HOW M J, et al. Polarisation vision: overcoming challenges of working with a property of light we barely see[J]. The Science of Nature, 2018, 105(3): 27. DOI
[3]	田柳, 高隽, 范之国, 等. 基于大气偏振模式分布规律的导航方向角计算方法[J]. 电子学报, 2012, 40(1): 141-146. DOI
	TIAN L, GAO J, FAN Z G, et al. The method of the navigation direction angle calculation with the distribution of the atmospheric polarization pattern[J]. Acta Electronica Sinica, 2012, 40(1): 141-146. (in Chinese) DOI
[4]	褚金奎, 关乐, 李世奇, 等. 大气偏振模式图分布及仿生偏振导航技术[J]. 遥感学报, 2018, 22(6): 969-979.
	CHU J K, GUAN L, LI S Q, et al. Atmospheric polarization field pattern distribution and polarization navigation technology[J]. Journal of Remote Sensing, 2018, 22(6): 969-979. (in Chinese)
[5]	范之国, 徐超, 吴川, 等. 大气偏振模式特征及其在自主导航中的应用[J]. 现代防御技术, 2017, 45(3): 1-7.
	FAN Z G, XU C, WU C, et al. Characteristics of atmospheric polarization pattern and its application in autonomous navigation[J]. Modern Defence Technology, 2017, 45(3): 1-7. (in Chinese)
[6]	XU Q, GUO Z Y, TAO Q Q, et al. Multi-spectral characteristics of polarization retrieve in various atmospheric conditions[J]. Optics Communications, 2015, 339: 167-170. DOI URL
[7]	YAN L, LI Y F, CHANDRASEKAR V, et al. General review of optical polarization remote sensing[J]. International Journal of Remote Sensing, 2020, 41(13): 4853-4864. DOI URL
[8]	晏磊, 关桂霞, 陈家斌, 等. 基于天空偏振光分布模式的仿生导航定向机理初探[J]. 北京大学学报: 自然科学版, 2009, 45(4): 616-620.
	YAN L, GUAN G X, CHEN J B, et al. The bionic orientation mechanism in the skylight polarization pattern[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2009, 45(4): 616-620. (in Chinese)
[9]	吴良海, 张骏, 范之国, 等. 多次散射因素影响下天空偏振光模式的解析模型[J]. 物理学报, 2014, 63(11): 114201.
	WU L H, ZHANG J, FAN Z G, et al. An analytical model for skylight polarization pattern with multiple scattering[J]. Acta Physica Sinica, 2014, 63(11): 114201. (in Chinese) DOI URL
[10]	赵开春, 卢皓, 尤政. 天空光偏振模式自动探测装置[J]. 光学精密工程, 2013, 21(2): 239-245.
	ZHAO K C, LU H, YOU Z. Automatic detection system for skylight polarized pattern[J]. Optics and Precision Engineering, 2013, 21(2): 239-245. (in Chinese) DOI URL
[11]	陈伟, 李延飞, 吴太夏, 等. 全天空偏振模式及其影响因素初探[J]. 遥感学报, 2018, 22(6): 989-995.
	CHEN W, LI Y F, WU T X, et al. Preliminary research on sky polarization and influential factors[J]. Journal of Remote Sensing, 2018, 22(6): 989-995. (in Chinese)
[12]	KONG Z, MA T, CHEN K, et al. Three-wavelength polarization Scheimpflug lidar system developed for remote sensing of atmospheric aerosols[J]. Applied Optics, 2019, 58(31): 8612-8621. DOI URL
[13]	魏昊明, 赵威, 戴星灿. 雾和气溶胶前向散射对消光的影响[J]. 光学精密工程, 2018, 26(6): 1354-1361.
	WEI H M, ZHAO W, DAI X C. Influence of fog and aerosol particles’ forward-scattering on light extinction[J]. Optics and Precision Engineering, 2018, 26(6): 1354-1361. (in Chinese) DOI URL
[14]	UGOLNIKOV O S, POSTYLYAKOV O V, MASLOV I A. Effects of multiple scattering and atmospheric aerosol on the polarization of the twilight sky[J]. Journal of Quantitative Spectroscopy and Radiative Transfer, 2004, 88(1-3): 233-241. DOI URL
[15]	强振平, 何丽波, 陈旭, 等. 深度学习图象修复方法综述[J]. 中国图象图形学报, 2019, 24(3): 447-463.
	QIANG Z P, HE L B, CHEN X, et al. Survey on deep learning image inpainting methods[J]. Jurnal of Image and Graphics, 2019, 24(3): 447-463. (in Chinese)
[16]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2536-2544.
[17]	LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 89-105.
[18]	YU J H, LIN Z, YANG J M, et al. Generative image inpainting with contextual attention[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5505-5514.
[19]	LI J Y, WANG N, ZHANG L F, et al. Recurrent feature reasoning for image inpainting[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7757-7765.
[20]	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[21]	甘鑫, 高欣健, 钟彬彬, 等. 基于有限样本的大气偏振模式生成方法[J]. 光电工程, 2021, 48(5): 200331.
	GAN X, GAO X J, ZHONG B B, et al. A few-shot learning based generative method for atmospheric polarization modelling[J]. Opto-Electronic Engineering, 2021, 48(5): 13-27. (in Chinese)
[22]	赵露露, 沈玲, 洪日昌. 图像修复研究进展综述[J]. 计算机科学, 2021, 48(3): 14-26. DOI
	ZHAO L L, SHEN L, HONG R C. Survey on image inpainting research progress[J]. Computer Science, 2021, 48(3): 14-26. (in Chinese)
[23]	程前, 高欣健, 高隽, 等. 基于邻域约束的大气偏振模式生成网络[J]. 光电工程, 2022, 49(6): 55-69.
	CHENG Q, GAO X J, GAO J, et al. A generative method for atmospheric polarization modelling based on neighborhood constraint[J]. Opto-Electronic Engineering, 2022, 49(6): 55-69. (in Chinese)
[24]	LEE S, OH S W, WON D, et al. Copy-and-paste networks for deep video inpainting[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 4412-4420.
[25]	YI Z L, TANG Q, AZIZI S, et al. Contextual residual aggregation for ultra high-resolution image inpainting[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7505-7514.
[26]	HUANG T, LI S J, JIA X, et al. Neighbor2Neighbor: self-supervised denoising from single noisy images[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 14776-14785.
[27]	LI A, ZHAO S S, MA X J, et al. Short-term and long-term context aggregation network for video inpainting[C]// European Conference on Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 728-743.
[28]	PENG J L, LIU D, XU S C, et al. Generating diverse structure for image inpainting with hierarchical VQ-VAE[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10770-10779.
[29]	LAVIGNE D A, BRETON M, FOURNIER G, et al. A new passive polarimetric imaging system collecting polarization signatures in the visible and infrared bands[EB/OL]. [2022-08-20]. https:spie.org/Publications/Proceedings/Paper/10.1117/12.819011?ss0=1.
[30]	DANG T Y, LIU Y T, GAO X J, et al. Multi-scale spatial transform network for atmospheric polarization prediction[C]// International Conference on Image and Graphics - ICIG 2021. Cham: Springer International Publishing, 2021: 479-490.