Research on cyclic generative network oriented to inter-layer interpolation of medical images

doi:10.11996/JG.j.2095-302X.2023030502

Abstract

Abstract:

Due to the limitations of imaging equipment performance and radiation dose, the inter-layer resolution of CT and MRI image sequences is considerably lower than the intra-layer resolution, which greatly impedes the application of medical images. Consequently, the problem of effectively improving the inter-layer resolution of medical image has become a critical concern. To address this problem, a cyclic generative network was proposed for inter-layer interpolation of medical images. The network mapped medical images into their corresponding binary images to achieve simple and fluent inter-layer interpolation processing for consecutive medical image sequences. The proposed network was composed of two modules. On the one hand, the image transformation module was designed with a generator sub-network containing 9 residual blocks and 2 bilinear upsampling modules to achieve effective image transformation. Then, the cyclic mapping between medical images and their corresponding binary images was achieved by the bidirectional nonlinear mapping capability learned by the module. On the other hand, the interpolation module combined motion estimation and image generation into a single convolution step and constructed a binary image Charbonnier difference loss function suitable for medical image features to further improve image resolution and perform interpolation processing for the binary image sequences. Experimental results on five multi-type datasets demonstrated that the proposed method surpassed advanced comparison methods in terms of the average signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) of the generated images, and was also better at handling detailed information such as image edges and contours.

Key words: inter-layer interpolation of medical image, cyclic generative network, generator, motion estimation, loss function

CLC Number:

TP391

SUN Long-fei, LIU Hui, YANG Feng-chang, LI Pan. Research on cyclic generative network oriented to inter-layer interpolation of medical images[J]. Journal of Graphics, 2023, 44(3): 502-512.

Figures/Tables 17

Fig. 1 Structure of the cyclic generative network

Fig. 2 Framework of the image transformation module ((a) Cyclic network structure; (b) Mapping process of binary image; (c) Inverse mapping process of medical image)

Fig. 3 Structure of the interpolation network

Fig. 4 Performance comparison of different methods

Table 1 Information of experimental datasets

数据来源	数据分类	数据类别	图像分辨率	层间距 (mm)
自建数据集	心脏	CT	512×512	2.5
	脑部	MRI	181×217	2.5
	肺部	CT	812×662	5.0
公共数据集	D-Lung	CT	648×486	5.0
公共数据集	K-Lung	CT	512×512	1.0

Fig. 5 Sample examples from ((a) K-Lung; (b) Private dataset; (c) D-Lung)

Fig. 6 Generation results comparison based on two modules respectively ((a) Transposed conv; (b) Bilinear upsampling module)

Table 2 Quantitative results under different values of α

α	PSNR (dB)	SSIM
7	30.863 1	0.917 9
8	32.584 7	0.930 2
9	33.225 5	0.934 4
10	33.540 8	0.936 6
11	33.382 0	0.934 1
12	32.439 4	0.929 5

Table 3 Quantitative results under different values of β

β	PSNR (dB)	SSIM
2	31.201 9	0.919 2
3	32.673 7	0.929 4
4	33.280 0	0.935 7
5	33.540 8	0.936 6
6	33.349 7	0.934 2
7	32.711 3	0.930 8

Table 4 Interpolation results based on different loss functions

损失函数	PSNR (dB)	SSIM
L₁	32.565 1	0.929 6
L₂	30.813 6	0.917 8
L_c	32.483 7	0.929 8
L_{1_}_f	33.082 0	0.933 4
L_{2_}_f	32.940 1	0.931 5
L_robust	33.540 8	0.936 6

Table 5 Interpolation results of the interpolation module and the cyclic generative network (PSNR/SSIM)

数据集	插值模块	循环生成网络
心脏	31.684 7/0.918 2	32.902 3/0.928 6
脑部	32.322 9/0.942 0	33.711 0/0.959 7
肺部	31.704 9/0.915 3	32.679 6/0.928 2
D-Lung	31.676 5/0.914 1	32.584 7/0.920 4
K-Lung	31.691 0/0.917 2	32.774 5/0.936 4

Fig. 7 Interpolation results comparison of the interpolation module and the cyclic generative network ((a) Ground-truth; (b) The interpolation module; (c) The cyclic generative network)

Fig. 8 Interpolation results comparison based on CycleGAN and the image transformation module ((a) Ground-truth; (b) CycleGAN; (c) The image transformation module)

Fig. 9 Comparison of the interpolation performance of different interpolation methods on the K-Lung dataset with 1 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

Fig. 10 Comparison of the interpolation performance of different interpolation methods on the private head dataset with 2.5 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

Fig. 11 Comparison of the interpolation performance of different interpolation methods on the private heart dataset with 5 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

Table 6 Quantitative comparison results of different interpolation methods on fixed datasets

算法网络	PSNR/SSIM
算法网络	心脏	脑部	肺部	D-Lung	K-Lung	平均	参数量(M)
SepConv_L1	31.084 7 0.914 7	31.910 8 0.940 1	31.201 3 0.910 8	31.146 1 0.907 2	31.128 1 0.911 5	31.294 2 0.916 9	21.6
SepConv_Lf	30.339 0 0.910 2	31.707 6 0.938 8	30.504 7 0.904 9	30.526 5 0.904 6	30.463 7 0.905 8	30.708 3 0.912 9	21.6
pix2pix	31.854 6 0.9207	32.627 7 0.952 1	32.047 6 0.917 5	32.084 7 0.916 8	32.010 5 0.920 0	32.125 0 0.925 4	44.6
CyclicGen	32.161 3 0.923 9	32.794 8 0.954 3	32.276 6 0.920 4	32.181 0 0.918 1	32.372 2 0.929 2	32.357 2 0.929 2	19.8
CAIN	32.550 5 0.925 5	33.260 4 0.956 8	32.404 8 0.923 9	32.480 2 0.919 4	32.580 9 0.934 8	32.655 4 0.932 1	42.8
VFIT	32.874 3 0.928 3	33.680 2 0.959 9	32.610 4 0.927 7	32.522 6 0.920 3	32.820 7 0.936 6	32.901 6 0.934 6	29.0
Proposed	32.902 3 0.928 6	33.711 0 0.959 7	32.679 6 0.928 2	32.584 7 0.920 4	32.774 5 0.936 4	32.930 4 0.934 7	30.9

References 29

[1]	SIM H, OH J, KIM M. XVFI: eXtreme video frame interpolation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 14469-14478.
[2]	林毓秀, 刘慧, 刘绍玲, 等. 三维CT层间图像超分辨重建与修复[J]. 计算机辅助设计与图形学学报, 2020, 32(6): 919-929.
	LIN Y X, LIU H, LIU S L, et al. Super-resolution reconstruction and inpainting of inter-layer image in 3D CT[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(6): 919-929. (in Chinese)
[3]	MEYER S, WANG O, ZIMMER H, et al. Phase-based frame interpolation for video[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1410-1418.
[4]	EHRHARDT J, SäRING D, HANDELS H. Optical flow based interpolation of temporal image sequences[C]// Medical Imaging 2006:Image Processing. Bellingham: SPIE Press. 2006: 830-837.
[5]	ZHANG W W, BRADY J M, BECHER H, et al. Spatio-temporal (2D+T) non-rigid registration of real-time 3D echocardiography and cardiovascular MR image sequences[J]. Physics in Medicine and Biology, 2011, 56(5): 1341-1360. DOI PMID
[6]	NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive convolution[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2270-2279.
[7]	NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive separable convolution[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 261-270.
[8]	ZHANG L, KARANI N, TANNER C, et al. Temporal interpolation via motion field prediction[EB/OL]. [2022-06-03]. https://arxiv.org/abs/1804.04440.
[9]	KARANI N, TANNER C, KOZERKE S, et al. Temporal interpolation of abdominal MRIs acquired during free-breathing[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International publishing, 2017: 359-367.
[10]	LIU Y L, LIAO Y T, LIN Y Y, et al. Deep video frame interpolation using cyclic frame generation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 8794-8802. DOI URL
[11]	CHOI M, KIM H, HAN B, et al. Channel attention is all You need for video frame interpolation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10663-10671. DOI URL
[12]	SHI Z H, XU X Y, LIU X H, et al. Video frame interpolation transformer[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 17461-17470.
[13]	YU H, YANG L T, ZHANG Q C, et al. Convolutional neural networks for medical image analysis: state-of-the-art, comparisons, improvement and perspectives[J]. Neurocomputing, 2021, 444: 92-110. DOI URL
[14]	李翠云, 白静, 郑凉. 融合边缘增强注意力机制和U-Net网络的医学图像分割[J]. 图学学报, 2022, 43(2): 273-278.
	LI C Y, BAI J, ZHENG L. A U-Net based contour enhanced attention for medical image segmentation[J]. Journal of Graphics, 2022, 43(2): 273-278. (in Chinese)
[15]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// The 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680.
[16]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[17]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5967-5976.
[18]	廖仕敏, 刘仰川, 朱叶晨, 等. 一种基于CycleGAN改进的低剂量CT图像增强网络[J]. 图学学报, 2022, 43(4): 570-579.
	LIAO S M, LIU Y C, ZHU Y C, et al. An improved low-dose CT image enhancement network based on CycleGAN[J]. Journal of Graphics, 2022, 43(4): 570-579. (in Chinese)
[19]	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]// European Conference on Computer Vision. Cham: Springer International publishing, 2016: 694-711.
[20]	TAIGMAN Y, POLYAK A, WOLF L. Unsupervised cross-domain image generation[EB/OL]. [2022-06-26]. https://arxiv.org/abs/1611.02200.
[21]	BARRON J T. A more general robust loss function[EB/OL]. [2022-07-02]. https://arxiv.org/pdf/1701.03077v2.pdf.
[22]	MATHIEU M, COUPRIE C, LECUN Y. Deep multi-scale video prediction beyond mean square error[EB/OL]. [2022-07-22]. https://arxiv.org/pdf/1511.05440.pdf.
[23]	GU J Q, LI Z J, WANG Y Y, et al. Deep generative adversarial networks for thin-section infant MR image reconstruction[J]. IEEE Access, 2019, 7: 68290-68304. DOI URL
[24]	白亮, 刘辉, 尚振宏. 自适应融合层级特征的混合退化图像复原算法[J]. 计算机辅助设计与图形学学报, 2021, 33(2): 215-222.
	BAI L, LIU H, SHANG Z H. Mixed degraded image restoration algorithm based on adaptive fusion of hierarchical features[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(2): 215-222. (in Chinese)
[25]	CASTILLO R, CASTILLO E, GUERRA R, et al. A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets[J]. Physics in Medicine and Biology, 2009, 54(7): 1849-1870. DOI PMID
[26]	YAN K, WANG X S, LU L, et al. DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning[J]. Journal of Medical Imaging, 2018, 5(3): 036501.
[27]	MAO X D, LI Q, XIE H R, et al. Least Squares generative adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2813-2821.
[28]	ODENA A, DUMOULIN V, OLAH C. Deconvolution and checkerboard artifacts[J]. Distill, 2016, 1(10): e3.
[29]	LI X S, ZHOU F Q, TAN H S, et al. Multimodal medical image fusion based on joint bilateral filter and local gradient energy[J]. Information Sciences, 2021, 569: 302-325. DOI URL