基于多尺度特征融合的细胞计数方法

doi:10.11996/JG.j.2095-302X.2023010041

摘要/Abstract

摘要：

细胞计数一直是医学影像分析中非常重要的一项工作，在生物医学实验和临床医学等领域起着十分关键的作用。针对细胞计数工作中存在的由细胞尺寸变化等因素造成的细胞计数精度低的问题，引入高度拥挤目标识别网络CSRNet并加以改进，构建了一种基于多尺度特征融合的细胞计数方法。首先，使用VGG16的前10层提取细胞特征，避免了由于网络过深造成的小目标信息丢失；其次，引入空间金字塔池化结构提取细胞的多尺度特征并进行特征融合，降低了因细胞形态各异、尺寸不一和细胞遮挡等问题带来的计数误差；然后，使用混合空洞卷积对特征图进行解码，得到密度图，解决了CSRNet在解码过程中像素遗漏的问题；最后对密度图逐像素进行回归得到细胞总数。另外，在训练过程中引入了一种新的组合损失函数以代替欧几里得损失函数，不仅考虑了ground truth密度图与预测密度图单个像素点之间的关系，还考虑了其全局和局部的密度水平。实验证明，优化后的CSRNet在VGG cells和MBM cells数据集上取得了较好的结果，有效改善了由细胞尺寸变化等因素造成的细胞计数精度低的问题。

关键词: 细胞计数, 多尺度特征融合, 密度估计, 空间金字塔池化, 混合空洞卷积

Abstract:

To address the problem of low cell counting accuracy caused by factors such as cell size variation in cell counting work, the highly crowded target recognition network CSRNet was introduced and improved, and a cell counting method based on multi-scale feature fusion was constructed. First, the first 10 layers of VGG16 were employed to extract cell features, avoiding the loss of small target information due to the deep network. Then, the spatial pyramid pooling structure was introduced to extract the multi-scale features of cells and perform feature fusion, reducing the counting errors caused by different cell shapes, sizes, and cell occlusion. Then the feature map was decoded using the hybrid dilated convolution to obtain the density map, solving the problem of missing pixels in the decoding process of CSRNet. Finally, the density map was regressed pixel by pixel to obtain the total number of cells. In addition, a new combined loss function was introduced in the training process to replace the Euclidean loss function, which not only considered the relationship between the ground truth density map and the single pixel point of the predicted density map, but also considered the global and local density levels. Experiments show that the optimized CSRNet could yield better results on VGG cells and MBM cells datasets, effectively improving the low cell counting accuracy caused by factors such as cell size variation.

Key words: cell counting, multi-scale feature fusion, density estimation, spatial pyramid pooling, hybrid dilated convolution

中图分类号:

TP391

张倩, 王夏黎, 王炜昊, 武历展, 李超. 基于多尺度特征融合的细胞计数方法[J]. 图学学报, 2023, 44(1): 41-49.

ZHANG Qian, WANG Xia-li, WANG Wei-hao, WU Li-zhan, LI Chao. Cell counting method based on multi-scale feature fusion[J]. Journal of Graphics, 2023, 44(1): 41-49.

图/表 16

图1 本文网络结构图

Fig. 1 Network structure diagram of this paper

表1 网络参数设置

Table 1 Network parameter configuration

网络	Layer	Parameters
Feature extraction module	1~2	Conv3-64-1
	1~2	Max pooling
	3~4	Conv3-128-1
	3~4	Max pooling
	5~7	Conv3-256-1
	5~7	Max pooling
	8~10	Conv3-512-1
Feature fusion module	S1	Avg pooling
	S1	Conv1-512-1
	S2	Avg pooling
	S2	Conv1-512-1
	S3	Avg pooling
	S3	Conv1-512-1
	S6	Avg pooling
	S6	Conv1-512-1
Feature decoding module	1	Conv3-512-1
	2	Conv3-512-2
	3	Conv3-512-3
	4	Conv3-256-1
	5	Conv3-128-2
	6	Conv3-64-3
Output	1	Conv1-1-1

图2 SPP结构图

Fig. 2 SPP structure diagram

图3 相反特征

Fig. 3 Contrary features

图4 空洞卷积的比较结果

Fig. 4 Comparison results of dialted convolution ((a) Dilation rate=[2,2,2]; (b) Dilation rate=[1,2,3])

图5 VGG cells示例图片

Fig. 5 Example image of VGG cells

图6 MBM cells示例图片

Fig. 6 Example image of MBM cells

图7 细胞图像及其对应的标注信息和ground truth密度图((a)细胞图像；(b)标注信息；(c) Ground truth密度图)

Fig. 7 Cell images and their corresponding annotations and ground truth density maps ((a) Cell image; (b) Annotation information; (c) Ground truth density map)

图8 损失函数

Fig. 8 Loss function

图9 VGG cells数据集计数结果((a)输入图像；(b) Ground truth 密度图；(c)预测密度图)

Fig. 9 Counting results of VGG cells dataset ((a) Input image; (b) Ground truth density map; (c) Predicted density map)

表2 VGG cells数据集比较结果

Table 2 Comparison results of VGG cells dataset

方法	MAE
方法	Ntrain=8	Ntrain=16	Ntrain=32	Ntrain=50
文献[14]	4.9±0.7	3.8±0.2	3.5±0.2	N/A
文献[15]	3.4±0.1	N/A	3.2±0.1	N/A
文献[18]	3.9±0.5	3.4±0.2	2.9±0.2	2.9±0.2*
文献[17]	N/A	N/A	N/A	2.6±0.4*
文献[15]	3.9±0.4	2.9±0.5	2.4±0.4	2.3±0.4
文献[27]	2.9±0.2	2.8±0.1	2.6±0.1	2.6±0.1
CSRNet	3.5±0.3	3.3±0.3	3.2±0.2	3.2±0.2
本文方法	3.1±0.2	2.9±0.2	2.8±0.2	2.8±0.1

表3 MBM cells数据集比较结果

Table 3 Comparison results of MBM cells dataset

方法	MAE
方法	Ntrain=5	Ntrain=10	Ntrain=15
文献[18]	28.9±22.6	22.2±11.6	21.3±9.4
文献[13]	23.6±4.6	21.5±4.2	20.5±3.5
文献[5]	12.6±3.0	10.7±2.5	8.8±2.3
文献[27]	9.3±1.4	8.9±0.9	8.6±0.3
CSRNet	9.7±1.3	8.9±0.8	8.2±0.5
本文方法	8.8±0.9	7.7±0.7	7.1±0.4

图10 MBM cells数据集计数结果((a)输入图像；(b) Ground truth 密度图；(c)预测密度图)

Fig. 10 Counting results of MBM cells dataset ((a) Input image; (b) Ground truth density map; (c) Predicted density map)

图11 本文方法与CSRNet在VGG cells数据集上的对比结果((a)输入图像；(b) Ground truth密度图；(c)本文方法预测密度图；(d) CSRNet预测密度图)

Fig. 11 Comparison of the method in this paper and CSRNet on the VGG cells dataset ((a) Input image; (b) Ground truth density map; (c) Density map predicted by this paper; (d) Density map predicted by CSRNet)

图12 本文方法与CSRNet在MBM cells数据集上的对比结果((a)输入图像；(b) Ground truth密度图；(c)本文方法预测密度图；(d) CSRNet预测密度图)

Fig. 12 Comparison of the method in this paper and CSRNet on the MBM cells dataset ((a) Input image; (b) Ground truth density map; (c) Density map predicted by this paper; (d) Density map predicted by CSRNet)

表4 MBM cells数据集消融研究结果

Table 4 Ablation study results of MBM cells dataset

方法	MAE
方法	Ntrain=5	Ntrain=10	Ntrain=15
M1	9.2±1.2	7.9±1.0	7.8±0.5
M2	8.9±1.0	7.8±0.7	7.2±0.4
M3	9.1±1.0	8.0±0.8	7.5±0.6
本文方法	8.8±0.9	7.7±0.7	7.1±0.4

参考文献 27

[1]	MAUSUMI M, RAHUL K G, MANALI M. Detection and counting of red blood cells in blood cell images using Hough transform[J]. International Journal of Computer Applications, 2012, 53(16): 13-17.
[2]	FAUSTINO G M, GATTASS M, REHEN S, et al. Automatic embryonic stem cells detection and counting method in fluorescence microscopy images[C]//2009 IEEE International Symposium on Biomedical Imaging:from Nano to Macro. New York: IEEE Press, 2009: 799-802.
[3]	SHARIF J M, MISWAN M F, NGADI M A, et al. Red blood cell segmentation using masking and watershed algorithm: a preliminary study[J]. 2012 International Conference on Biomedical Engineering, 2012, 2012: 258-262.
[4]	XUE Y, RAY N, HUGH J, et al. Cell counting by regression using convolutional neural network[C]// European Conference on Computer Vision. Chma: Springer International Publishing, 2016: 274-290.
[5]	COHEN J P, BOUCHER G, GLASTONBURY C A, et al. Count-ception: counting by fully convolutional redundant counting[C]//2017 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2018: 18-26.
[6]	JIANG N, YU F H. A multi-column network for cell counting[J]. OSA Continuum, 2020, 3(7): 1834-1846. DOI URL
[7]	LI Y, ZHANG X, CHEN D. CSRNet: dilated convolutional neural networks for understanding the highly congested scenes[EB/OL]. (2018-04-11) [2021-12-08].https://arxiv.org/abs/1802.10062.
[8]	ZHANG C, SUN C, SU R, et al. Segmentation of clustered nuclei based on curvature weighting[C]// The 27th Conference on Image Vision Computing. New York: ACM, 2012: 49.
[9]	刘树杰. 基于卷积神经网络的红细胞检测和计数方法[D]. 广州: 华南理工大学, 2017.
	LIU S J. Red blood cell detection and counting based on convolutional neural network[D]. Guangzhou: South China University of Technology, 2017 (in Chinese).
[10]	徐晓涛. 基于深度目标识别的细胞计数研究[D]. 合肥: 安徽大学, 2020.
	XU X T. Research on cell counting based on depth target recognition[D]. Hefei: Anhui University, 2020 (in Chinese).
[11]	洪羽萌. 外周血细胞分类与计数在深度学习中的应用[D]. 南京: 南京大学, 2019.
	HONG Y M. Application of peripheral blood cell classification and counting in deep learning[D]. Nanjing: Nanjing University, 2019 (in Chinese).
[12]	AICH S, STAVNESS I. Improving object counting with heatmap regulation[EB/OL]. (2018-03-14) [2021-12-08]. https://www.semanticscholar.org/paper/Improving-Object-Counting-with-Heatmap-Regulation-Aich-Stavness/aabbc2880cdc9af83322e2bac8e2f88d1105f601.
[13]	MARSDEN M, MCGUINNESS K, LITTLE S, et al. People, penguins and petri dishes: adapting object counting models to new visual domains and object types without forgetting[EB/OL]. (2017-11-15) [2022-01-03]. https://ieeexplore.ieee.org/document/8578940.
[14]	LEMPITSKY V, ZISSERMAN A. Learning to count objects in images[EB/OL]. (2010-12-06) [2021-12-18]. https://dl.acm.org/doi/10.5555/2997189.2997337.
[15]	FIASCHI L, KOETHE U, NAIR R, et al. Learning to count with regression forest and structured labels[EB/OL]. (2012-11-01) [2021-12-22]. https://ieeexplore.ieee.org/document/6460719.
[16]	JIANG N, YU F. A cell counting framework based on random forest and density map[J]. Applied Sciences, 2020, 10(23): 8346. DOI URL
[17]	GUO Y, STEIN J, WU G, et al. SAU-Net: a universal deep network for cell counting[C]//The 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. New York: ACM, 2019: 299-306.
[18]	XIE W, NOBLE J A, ZISSERMAN A. Microscopy cell counting and detection with fully, convolutional regression networks[J]. Computer Methods in Biomechanics and Bio, 2018, 6(3-4): 283-292.
[19]	HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID
[20]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-09-04) [2022-01-10]. https://arxiv.org/abs/1409.1556v6.
[21]	LIU W, SALZMANN M, FUA P. Context-aware crowd counting[EB/OL]. (2019-04-15) [2021-12-13]. https://infoscience.epfl.ch/record/265758.
[22]	WANG P, CHEN P, YUAN Y, et al. Understanding convolution for semantic segmentation[EB/OL]. (2018-06-01) [2021-12-28]. https://arxiv.org/abs/1702.08502v3.
[23]	KAINZ P, URSCHLER M, SCHULTER S, et al. You should use regression to detect cells[C]//International Conference on Medical Image Computing and Computer Assisted Intervention-MICCAI. Heidelberg:Springer, 2015: 276-283.
[24]	ZHANG Y, ZHOU D, CHEN S, et al. Single-image crowd counting via multi-column convolutional neural network[C]//2016 IEEE Conf- erence on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 589-597
[25]	JIANG X, XIAO Z, ZHANG B, et al. Crowd counting and density estimation by trellis encoder-decoder networks[EB/OL]. [2021-12-28]. https://xueshu.baidu.com/usercenter/paper/show?paperid=147606t0n15c0860jb0b0010hr729346.
[26]	DAI F, LIU H, MA Y, et al. Dense scale network for crowd counting[EB/OL]. (2021-09-01) [2022-12-28]. https://dl.acm.org/doi/10.1145/3460426.3463628.
[27]	蒋妮. 基于密度估计的细胞计数方法及其在显微图像中的应用研究[D]. 杭州: 浙江大学, 2021.
	JIANG N. Research on cell counting based on density estimation and its applications in microscopic image[D]. Hangzhou: Zhejiang University, 2021 (in Chinese).