面向医学图像层间插值的循环生成网络研究

doi:10.11996/JG.j.2095-302X.2023030502

图学学报 ›› 2023, Vol. 44 ›› Issue (3): 502-512.DOI: 10.11996/JG.j.2095-302X.2023030502

• 图像处理与计算机视觉 • 上一篇下一篇

面向医学图像层间插值的循环生成网络研究

孙龙飞¹^,²(), 刘慧¹^,²(), 杨奉常³, 李攀⁴

1.山东财经大学计算机科学与技术学院，山东济南 250014
2.山东省数字媒体技术重点实验室，山东济南 250014
3.山东第一医科大学附属肿瘤医院，山东济南 250014
4.重庆医科大学附属第二医院，重庆 400010

收稿日期:2022-10-07 接受日期:2022-12-17 出版日期:2023-06-30 发布日期:2023-06-30
通讯作者: 刘慧(1978-)，女，教授，博士。主要研究方向为数据挖掘与可视化。E-mail：liuh_lh@sdufe.edu.cn
作者简介:
孙龙飞(1996-)，男，硕士研究生。主要研究方向为医学图像处理。E-mail：slongfei2021@163.com
基金资助:
国家自然科学基金项目(62072274);重庆市科技局鲁渝科技协作项目(cstc2021jscx-lyjsAX0003);中央引导地方科技发展项目(YDZX2022009)

Research on cyclic generative network oriented to inter-layer interpolation of medical images

SUN Long-fei¹^,²(), LIU Hui¹^,²(), YANG Feng-chang³, LI Pan⁴

1. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan Shandong 250014, China
2. Shandong Key Laboratory of Digital Media Technology, Jinan Shandong 250014, China
3. Cancer Hospital Affiliated to Shandong First Medical University, Jinan Shandong 250014, China
4. The Second Affiliated Hospital of Chongqing Medical University, Chongqing 400010, China

Received:2022-10-07 Accepted:2022-12-17 Online:2023-06-30 Published:2023-06-30
Contact: LIU Hui (1978-), professor, Ph.D. Her main research interest covers data mining and visualization. E-mail：liuh_lh@sdufe.edu.cn
About author:
SUN Long-fei (1996-), master student. His main research interest covers medical image processing. E-mail：slongfei2021@163.com
Supported by:
National Natural Science Foundation of China(62072274);Chongqing Science and Technology Bureau Lu-Yu Science and Technology Collaboration Project(cstc2021jscx-lyjsAX0003);Central Guidance on Local Science and Technology Development Project(YDZX2022009)

摘要/Abstract

摘要：

由于受成像设备性能及辐射剂量等因素的限制，CT以及MRI图像序列的层间分辨率远低于层内分辨率，这极大地限制了医学图像的应用，如何有效提高医学图像序列的层间分辨率是一个亟待解决的问题。针对此问题，将医学图像转换成对应的二值图像，实现对连续医学图像序列简单且流畅的层间插值处理，提出一种医学图像层间插值循环生成网络。该网络由2个模块构成：图像转换模块设计包含9个残差块和2个双线性上采样模块的生成器子网络实现有效的图像转换，然后通过该模块学习到的双向非线性映射能力实现医学图像和其对应二值图像之间的循环映射；插值模块将运动估计和图像生成合并到单个卷积步骤中，并构造一个适于医学图像特征的二值图Charbonnier差损失函数进一步提高图像清晰度，完成对二值图像序列的插值处理。在5个多类型数据集上的实验结果表明，生成图像的平均峰值信噪比(PSNR)和结构相似性(SSIM)均优于先进对比方法，在图像边缘、轮廓等细节信息的处理上更加出色。

关键词: 医学图像层间插值, 循环生成网络, 生成器, 运动估计, 损失函数

Abstract:

Due to the limitations of imaging equipment performance and radiation dose, the inter-layer resolution of CT and MRI image sequences is considerably lower than the intra-layer resolution, which greatly impedes the application of medical images. Consequently, the problem of effectively improving the inter-layer resolution of medical image has become a critical concern. To address this problem, a cyclic generative network was proposed for inter-layer interpolation of medical images. The network mapped medical images into their corresponding binary images to achieve simple and fluent inter-layer interpolation processing for consecutive medical image sequences. The proposed network was composed of two modules. On the one hand, the image transformation module was designed with a generator sub-network containing 9 residual blocks and 2 bilinear upsampling modules to achieve effective image transformation. Then, the cyclic mapping between medical images and their corresponding binary images was achieved by the bidirectional nonlinear mapping capability learned by the module. On the other hand, the interpolation module combined motion estimation and image generation into a single convolution step and constructed a binary image Charbonnier difference loss function suitable for medical image features to further improve image resolution and perform interpolation processing for the binary image sequences. Experimental results on five multi-type datasets demonstrated that the proposed method surpassed advanced comparison methods in terms of the average signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) of the generated images, and was also better at handling detailed information such as image edges and contours.

Key words: inter-layer interpolation of medical image, cyclic generative network, generator, motion estimation, loss function

中图分类号:

TP391

孙龙飞, 刘慧, 杨奉常, 李攀. 面向医学图像层间插值的循环生成网络研究[J]. 图学学报, 2023, 44(3): 502-512.

SUN Long-fei, LIU Hui, YANG Feng-chang, LI Pan. Research on cyclic generative network oriented to inter-layer interpolation of medical images[J]. Journal of Graphics, 2023, 44(3): 502-512.

图/表 17

图1 循环生成网络结构

Fig. 1 Structure of the cyclic generative network

图2 图像转换模块架构((a)网络的环形结构；(b)二值图像映射过程；(c)医学图像反映射过程)

Fig. 2 Framework of the image transformation module ((a) Cyclic network structure; (b) Mapping process of binary image; (c) Inverse mapping process of medical image)

图3 插值网络结构

Fig. 3 Structure of the interpolation network

图4 不同方法性能对比

Fig. 4 Performance comparison of different methods

表1 实验数据集相关信息

Table 1 Information of experimental datasets

数据来源	数据分类	数据类别	图像分辨率	层间距 (mm)
自建数据集	心脏	CT	512×512	2.5
	脑部	MRI	181×217	2.5
	肺部	CT	812×662	5.0
公共数据集	D-Lung	CT	648×486	5.0
公共数据集	K-Lung	CT	512×512	1.0

图5 样本示例

Fig. 5 Sample examples from ((a) K-Lung; (b) Private dataset; (c) D-Lung)

图6 分别基于2模块的生成结果对比((a)转置卷积；(b)双线性上采样模块)

Fig. 6 Generation results comparison based on two modules respectively ((a) Transposed conv; (b) Bilinear upsampling module)

表2 在α不同取值下的定量结果

Table 2 Quantitative results under different values of α

α	PSNR (dB)	SSIM
7	30.863 1	0.917 9
8	32.584 7	0.930 2
9	33.225 5	0.934 4
10	33.540 8	0.936 6
11	33.382 0	0.934 1
12	32.439 4	0.929 5

表3 在β不同取值下的定量结果

Table 3 Quantitative results under different values of β

β	PSNR (dB)	SSIM
2	31.201 9	0.919 2
3	32.673 7	0.929 4
4	33.280 0	0.935 7
5	33.540 8	0.936 6
6	33.349 7	0.934 2
7	32.711 3	0.930 8

表4 基于不同损失函数的插值结果

Table 4 Interpolation results based on different loss functions

损失函数	PSNR (dB)	SSIM
L₁	32.565 1	0.929 6
L₂	30.813 6	0.917 8
L_c	32.483 7	0.929 8
L_{1_}_f	33.082 0	0.933 4
L_{2_}_f	32.940 1	0.931 5
L_robust	33.540 8	0.936 6

表5 插值模块和循环生成网络的插值结果(PSNR/SSIM)

Table 5 Interpolation results of the interpolation module and the cyclic generative network (PSNR/SSIM)

数据集	插值模块	循环生成网络
心脏	31.684 7/0.918 2	32.902 3/0.928 6
脑部	32.322 9/0.942 0	33.711 0/0.959 7
肺部	31.704 9/0.915 3	32.679 6/0.928 2
D-Lung	31.676 5/0.914 1	32.584 7/0.920 4
K-Lung	31.691 0/0.917 2	32.774 5/0.936 4

图7 插值模块和循环生成网络的插值结果对比((a) Ground-truth；(b)插值模块；(c)循环生成网络)

Fig. 7 Interpolation results comparison of the interpolation module and the cyclic generative network ((a) Ground-truth; (b) The interpolation module; (c) The cyclic generative network)

图8 基于CycleGAN和图像转换模块的插值结果对比((a) Ground-truth；(b) CycleGAN；(c)图像转换模块)

Fig. 8 Interpolation results comparison based on CycleGAN and the image transformation module ((a) Ground-truth; (b) CycleGAN; (c) The image transformation module)

图9 不同插值方法对1 mm层间距K-Lung数据集的插值性能对比

Fig. 9 Comparison of the interpolation performance of different interpolation methods on the K-Lung dataset with 1 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

图10 不同插值方法对2.5 mm层间距自建脑部数据集的插值性能对比

Fig. 10 Comparison of the interpolation performance of different interpolation methods on the private head dataset with 2.5 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

图11 不同插值方法对5 mm层间距自建心脏数据集的插值性能对比

Fig. 11 Comparison of the interpolation performance of different interpolation methods on the private heart dataset with 5 mm layer spacing ((a) Ground-truth; (b) Pix2pix; (c) CyclicGen; (d) CAIN; (e) VFIT; (f) Proposed)

表6 不同插值方法在固定数据集上的定量对比结果

Table 6 Quantitative comparison results of different interpolation methods on fixed datasets

算法网络	PSNR/SSIM
算法网络	心脏	脑部	肺部	D-Lung	K-Lung	平均	参数量(M)
SepConv_L1	31.084 7 0.914 7	31.910 8 0.940 1	31.201 3 0.910 8	31.146 1 0.907 2	31.128 1 0.911 5	31.294 2 0.916 9	21.6
SepConv_Lf	30.339 0 0.910 2	31.707 6 0.938 8	30.504 7 0.904 9	30.526 5 0.904 6	30.463 7 0.905 8	30.708 3 0.912 9	21.6
pix2pix	31.854 6 0.9207	32.627 7 0.952 1	32.047 6 0.917 5	32.084 7 0.916 8	32.010 5 0.920 0	32.125 0 0.925 4	44.6
CyclicGen	32.161 3 0.923 9	32.794 8 0.954 3	32.276 6 0.920 4	32.181 0 0.918 1	32.372 2 0.929 2	32.357 2 0.929 2	19.8
CAIN	32.550 5 0.925 5	33.260 4 0.956 8	32.404 8 0.923 9	32.480 2 0.919 4	32.580 9 0.934 8	32.655 4 0.932 1	42.8
VFIT	32.874 3 0.928 3	33.680 2 0.959 9	32.610 4 0.927 7	32.522 6 0.920 3	32.820 7 0.936 6	32.901 6 0.934 6	29.0
Proposed	32.902 3 0.928 6	33.711 0 0.959 7	32.679 6 0.928 2	32.584 7 0.920 4	32.774 5 0.936 4	32.930 4 0.934 7	30.9

参考文献 29

[1]	SIM H, OH J, KIM M. XVFI: eXtreme video frame interpolation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 14469-14478.
[2]	林毓秀, 刘慧, 刘绍玲, 等. 三维CT层间图像超分辨重建与修复[J]. 计算机辅助设计与图形学学报, 2020, 32(6): 919-929.
	LIN Y X, LIU H, LIU S L, et al. Super-resolution reconstruction and inpainting of inter-layer image in 3D CT[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(6): 919-929. (in Chinese)
[3]	MEYER S, WANG O, ZIMMER H, et al. Phase-based frame interpolation for video[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1410-1418.
[4]	EHRHARDT J, SäRING D, HANDELS H. Optical flow based interpolation of temporal image sequences[C]// Medical Imaging 2006:Image Processing. Bellingham: SPIE Press. 2006: 830-837.
[5]	ZHANG W W, BRADY J M, BECHER H, et al. Spatio-temporal (2D+T) non-rigid registration of real-time 3D echocardiography and cardiovascular MR image sequences[J]. Physics in Medicine and Biology, 2011, 56(5): 1341-1360. DOI PMID
[6]	NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive convolution[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2270-2279.
[7]	NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive separable convolution[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 261-270.
[8]	ZHANG L, KARANI N, TANNER C, et al. Temporal interpolation via motion field prediction[EB/OL]. [2022-06-03]. https://arxiv.org/abs/1804.04440.
[9]	KARANI N, TANNER C, KOZERKE S, et al. Temporal interpolation of abdominal MRIs acquired during free-breathing[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer International publishing, 2017: 359-367.
[10]	LIU Y L, LIAO Y T, LIN Y Y, et al. Deep video frame interpolation using cyclic frame generation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 8794-8802. DOI URL
[11]	CHOI M, KIM H, HAN B, et al. Channel attention is all You need for video frame interpolation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10663-10671. DOI URL
[12]	SHI Z H, XU X Y, LIU X H, et al. Video frame interpolation transformer[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 17461-17470.
[13]	YU H, YANG L T, ZHANG Q C, et al. Convolutional neural networks for medical image analysis: state-of-the-art, comparisons, improvement and perspectives[J]. Neurocomputing, 2021, 444: 92-110. DOI URL
[14]	李翠云, 白静, 郑凉. 融合边缘增强注意力机制和U-Net网络的医学图像分割[J]. 图学学报, 2022, 43(2): 273-278.
	LI C Y, BAI J, ZHENG L. A U-Net based contour enhanced attention for medical image segmentation[J]. Journal of Graphics, 2022, 43(2): 273-278. (in Chinese)
[15]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// The 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680.
[16]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[17]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5967-5976.
[18]	廖仕敏, 刘仰川, 朱叶晨, 等. 一种基于CycleGAN改进的低剂量CT图像增强网络[J]. 图学学报, 2022, 43(4): 570-579.
	LIAO S M, LIU Y C, ZHU Y C, et al. An improved low-dose CT image enhancement network based on CycleGAN[J]. Journal of Graphics, 2022, 43(4): 570-579. (in Chinese)
[19]	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]// European Conference on Computer Vision. Cham: Springer International publishing, 2016: 694-711.
[20]	TAIGMAN Y, POLYAK A, WOLF L. Unsupervised cross-domain image generation[EB/OL]. [2022-06-26]. https://arxiv.org/abs/1611.02200.
[21]	BARRON J T. A more general robust loss function[EB/OL]. [2022-07-02]. https://arxiv.org/pdf/1701.03077v2.pdf.
[22]	MATHIEU M, COUPRIE C, LECUN Y. Deep multi-scale video prediction beyond mean square error[EB/OL]. [2022-07-22]. https://arxiv.org/pdf/1511.05440.pdf.
[23]	GU J Q, LI Z J, WANG Y Y, et al. Deep generative adversarial networks for thin-section infant MR image reconstruction[J]. IEEE Access, 2019, 7: 68290-68304. DOI URL
[24]	白亮, 刘辉, 尚振宏. 自适应融合层级特征的混合退化图像复原算法[J]. 计算机辅助设计与图形学学报, 2021, 33(2): 215-222.
	BAI L, LIU H, SHANG Z H. Mixed degraded image restoration algorithm based on adaptive fusion of hierarchical features[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(2): 215-222. (in Chinese)
[25]	CASTILLO R, CASTILLO E, GUERRA R, et al. A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets[J]. Physics in Medicine and Biology, 2009, 54(7): 1849-1870. DOI PMID
[26]	YAN K, WANG X S, LU L, et al. DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning[J]. Journal of Medical Imaging, 2018, 5(3): 036501.
[27]	MAO X D, LI Q, XIE H R, et al. Least Squares generative adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2813-2821.
[28]	ODENA A, DUMOULIN V, OLAH C. Deconvolution and checkerboard artifacts[J]. Distill, 2016, 1(10): e3.
[29]	LI X S, ZHOU F Q, TAN H S, et al. Multimodal medical image fusion based on joint bilateral filter and local gradient energy[J]. Information Sciences, 2021, 569: 302-325. DOI URL

面向医学图像层间插值的循环生成网络研究

Research on cyclic generative network oriented to inter-layer interpolation of medical images

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 29

相关文章 13

编辑推荐

Metrics

本文评价

[1]	郝帅, 赵新生, 马旭, 张旭, 何田, 侯李祥. 基于 TR-YOLOv5 的输电线路多类缺陷目标检测方法 [J]. 图学学报, 2023, 44(4): 667-676.
[2]	李刚, 张运涛, 汪文凯, 张东阳. 采用DETR与先验知识融合的输电线路螺栓缺陷检测方法[J]. 图学学报, 2023, 44(3): 438-447.
[3]	熊举举 , 徐杨 , 范润泽 , 孙少聪 . 基于轻量化视觉 Transformer 的花卉识别 [J]. 图学学报, 2023, 44(2): 271-279.
[4]	黄志勇, 韩莎莎, 陈致君, 姚玉, 熊彪, 马凯. 一种用于视频对象分割的仿 U 形网络 [J]. 图学学报, 2023, 44(1): 104-111.
[5]	郭文, 李冬, 袁飞 . 多尺度注意力融合和抗噪声的轻量点云人脸识别模型[J]. 图学学报, 2022, 43(6): 1124-1133.
[6]	赵璐璐 , 王学营 , 张翼 , 张美月 . 基于 YOLOv5s 融合 SENet 的车辆目标检测技术研究[J]. 图学学报, 2022, 43(5): 776-782.
[7]	武历展, 王夏黎, 张倩, 王炜昊, 李超. 基于优化 YOLOv5s 的跌倒人物目标检测方法[J]. 图学学报, 2022, 43(5): 791-802.
[8]	胡海涛 , 杜昊晨 , 王素琴 , 石敏 , 朱登明 , . 改进 YOLOX 的药品泡罩铝箔表面缺陷检测方法[J]. 图学学报, 2022, 43(5): 803-814.
[9]	李妮妮, 王夏黎, 付阳阳, 郑凤仙, 何丹丹, 袁绍欣. 一种优化 YOLO 模型的交通警察目标检测方法[J]. 图学学报, 2022, 43(2): 296-305.
[10]	徐晨 , 倪蓉蓉 , 赵耀 , . 融合稀疏点云补全的 3D 目标检测算法[J]. 图学学报, 2021, 42(1): 37-43.
[11]	刘艳. 引入搜索预测与阈值决策的改进菱形运动估计算法[J]. 图学学报, 2015, 36(4): 576-580.
[12]	刘艳. 基于DS-CDS 的改进运动估计算法及实验[J]. 图学学报, 2015, 36(3): 457-561.
[13]	陈皓，霍星. 视频监控中人脸识别算法稳定性的改进[J]. 图学学报, 2011, 32(6): 53-56.