Cell counting method based on multi-scale feature fusion

doi:10.11996/JG.j.2095-302X.2023010041

Abstract

Abstract:

To address the problem of low cell counting accuracy caused by factors such as cell size variation in cell counting work, the highly crowded target recognition network CSRNet was introduced and improved, and a cell counting method based on multi-scale feature fusion was constructed. First, the first 10 layers of VGG16 were employed to extract cell features, avoiding the loss of small target information due to the deep network. Then, the spatial pyramid pooling structure was introduced to extract the multi-scale features of cells and perform feature fusion, reducing the counting errors caused by different cell shapes, sizes, and cell occlusion. Then the feature map was decoded using the hybrid dilated convolution to obtain the density map, solving the problem of missing pixels in the decoding process of CSRNet. Finally, the density map was regressed pixel by pixel to obtain the total number of cells. In addition, a new combined loss function was introduced in the training process to replace the Euclidean loss function, which not only considered the relationship between the ground truth density map and the single pixel point of the predicted density map, but also considered the global and local density levels. Experiments show that the optimized CSRNet could yield better results on VGG cells and MBM cells datasets, effectively improving the low cell counting accuracy caused by factors such as cell size variation.

Key words: cell counting, multi-scale feature fusion, density estimation, spatial pyramid pooling, hybrid dilated convolution

CLC Number:

TP391

ZHANG Qian, WANG Xia-li, WANG Wei-hao, WU Li-zhan, LI Chao. Cell counting method based on multi-scale feature fusion[J]. Journal of Graphics, 2023, 44(1): 41-49.

Figures/Tables 16

Fig. 1 Network structure diagram of this paper

Table 1 Network parameter configuration

网络	Layer	Parameters
Feature extraction module	1~2	Conv3-64-1
	1~2	Max pooling
	3~4	Conv3-128-1
	3~4	Max pooling
	5~7	Conv3-256-1
	5~7	Max pooling
	8~10	Conv3-512-1
Feature fusion module	S1	Avg pooling
	S1	Conv1-512-1
	S2	Avg pooling
	S2	Conv1-512-1
	S3	Avg pooling
	S3	Conv1-512-1
	S6	Avg pooling
	S6	Conv1-512-1
Feature decoding module	1	Conv3-512-1
	2	Conv3-512-2
	3	Conv3-512-3
	4	Conv3-256-1
	5	Conv3-128-2
	6	Conv3-64-3
Output	1	Conv1-1-1

Fig. 2 SPP structure diagram

Fig. 3 Contrary features

Fig. 4 Comparison results of dialted convolution ((a) Dilation rate=[2,2,2]; (b) Dilation rate=[1,2,3])

Fig. 5 Example image of VGG cells

Fig. 6 Example image of MBM cells

Fig. 7 Cell images and their corresponding annotations and ground truth density maps ((a) Cell image; (b) Annotation information; (c) Ground truth density map)

Fig. 8 Loss function

Fig. 9 Counting results of VGG cells dataset ((a) Input image; (b) Ground truth density map; (c) Predicted density map)

Table 2 Comparison results of VGG cells dataset

方法	MAE
方法	Ntrain=8	Ntrain=16	Ntrain=32	Ntrain=50
文献[14]	4.9±0.7	3.8±0.2	3.5±0.2	N/A
文献[15]	3.4±0.1	N/A	3.2±0.1	N/A
文献[18]	3.9±0.5	3.4±0.2	2.9±0.2	2.9±0.2*
文献[17]	N/A	N/A	N/A	2.6±0.4*
文献[15]	3.9±0.4	2.9±0.5	2.4±0.4	2.3±0.4
文献[27]	2.9±0.2	2.8±0.1	2.6±0.1	2.6±0.1
CSRNet	3.5±0.3	3.3±0.3	3.2±0.2	3.2±0.2
本文方法	3.1±0.2	2.9±0.2	2.8±0.2	2.8±0.1

Table 3 Comparison results of MBM cells dataset

方法	MAE
方法	Ntrain=5	Ntrain=10	Ntrain=15
文献[18]	28.9±22.6	22.2±11.6	21.3±9.4
文献[13]	23.6±4.6	21.5±4.2	20.5±3.5
文献[5]	12.6±3.0	10.7±2.5	8.8±2.3
文献[27]	9.3±1.4	8.9±0.9	8.6±0.3
CSRNet	9.7±1.3	8.9±0.8	8.2±0.5
本文方法	8.8±0.9	7.7±0.7	7.1±0.4

Fig. 10 Counting results of MBM cells dataset ((a) Input image; (b) Ground truth density map; (c) Predicted density map)

Fig. 11 Comparison of the method in this paper and CSRNet on the VGG cells dataset ((a) Input image; (b) Ground truth density map; (c) Density map predicted by this paper; (d) Density map predicted by CSRNet)

Fig. 12 Comparison of the method in this paper and CSRNet on the MBM cells dataset ((a) Input image; (b) Ground truth density map; (c) Density map predicted by this paper; (d) Density map predicted by CSRNet)

Table 4 Ablation study results of MBM cells dataset

方法	MAE
方法	Ntrain=5	Ntrain=10	Ntrain=15
M1	9.2±1.2	7.9±1.0	7.8±0.5
M2	8.9±1.0	7.8±0.7	7.2±0.4
M3	9.1±1.0	8.0±0.8	7.5±0.6
本文方法	8.8±0.9	7.7±0.7	7.1±0.4

References 27

[1]	MAUSUMI M, RAHUL K G, MANALI M. Detection and counting of red blood cells in blood cell images using Hough transform[J]. International Journal of Computer Applications, 2012, 53(16): 13-17.
[2]	FAUSTINO G M, GATTASS M, REHEN S, et al. Automatic embryonic stem cells detection and counting method in fluorescence microscopy images[C]//2009 IEEE International Symposium on Biomedical Imaging:from Nano to Macro. New York: IEEE Press, 2009: 799-802.
[3]	SHARIF J M, MISWAN M F, NGADI M A, et al. Red blood cell segmentation using masking and watershed algorithm: a preliminary study[J]. 2012 International Conference on Biomedical Engineering, 2012, 2012: 258-262.
[4]	XUE Y, RAY N, HUGH J, et al. Cell counting by regression using convolutional neural network[C]// European Conference on Computer Vision. Chma: Springer International Publishing, 2016: 274-290.
[5]	COHEN J P, BOUCHER G, GLASTONBURY C A, et al. Count-ception: counting by fully convolutional redundant counting[C]//2017 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2018: 18-26.
[6]	JIANG N, YU F H. A multi-column network for cell counting[J]. OSA Continuum, 2020, 3(7): 1834-1846. DOI URL
[7]	LI Y, ZHANG X, CHEN D. CSRNet: dilated convolutional neural networks for understanding the highly congested scenes[EB/OL]. (2018-04-11) [2021-12-08].https://arxiv.org/abs/1802.10062.
[8]	ZHANG C, SUN C, SU R, et al. Segmentation of clustered nuclei based on curvature weighting[C]// The 27th Conference on Image Vision Computing. New York: ACM, 2012: 49.
[9]	刘树杰. 基于卷积神经网络的红细胞检测和计数方法[D]. 广州: 华南理工大学, 2017.
	LIU S J. Red blood cell detection and counting based on convolutional neural network[D]. Guangzhou: South China University of Technology, 2017 (in Chinese).
[10]	徐晓涛. 基于深度目标识别的细胞计数研究[D]. 合肥: 安徽大学, 2020.
	XU X T. Research on cell counting based on depth target recognition[D]. Hefei: Anhui University, 2020 (in Chinese).
[11]	洪羽萌. 外周血细胞分类与计数在深度学习中的应用[D]. 南京: 南京大学, 2019.
	HONG Y M. Application of peripheral blood cell classification and counting in deep learning[D]. Nanjing: Nanjing University, 2019 (in Chinese).
[12]	AICH S, STAVNESS I. Improving object counting with heatmap regulation[EB/OL]. (2018-03-14) [2021-12-08]. https://www.semanticscholar.org/paper/Improving-Object-Counting-with-Heatmap-Regulation-Aich-Stavness/aabbc2880cdc9af83322e2bac8e2f88d1105f601.
[13]	MARSDEN M, MCGUINNESS K, LITTLE S, et al. People, penguins and petri dishes: adapting object counting models to new visual domains and object types without forgetting[EB/OL]. (2017-11-15) [2022-01-03]. https://ieeexplore.ieee.org/document/8578940.
[14]	LEMPITSKY V, ZISSERMAN A. Learning to count objects in images[EB/OL]. (2010-12-06) [2021-12-18]. https://dl.acm.org/doi/10.5555/2997189.2997337.
[15]	FIASCHI L, KOETHE U, NAIR R, et al. Learning to count with regression forest and structured labels[EB/OL]. (2012-11-01) [2021-12-22]. https://ieeexplore.ieee.org/document/6460719.
[16]	JIANG N, YU F. A cell counting framework based on random forest and density map[J]. Applied Sciences, 2020, 10(23): 8346. DOI URL
[17]	GUO Y, STEIN J, WU G, et al. SAU-Net: a universal deep network for cell counting[C]//The 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. New York: ACM, 2019: 299-306.
[18]	XIE W, NOBLE J A, ZISSERMAN A. Microscopy cell counting and detection with fully, convolutional regression networks[J]. Computer Methods in Biomechanics and Bio, 2018, 6(3-4): 283-292.
[19]	HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID
[20]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-09-04) [2022-01-10]. https://arxiv.org/abs/1409.1556v6.
[21]	LIU W, SALZMANN M, FUA P. Context-aware crowd counting[EB/OL]. (2019-04-15) [2021-12-13]. https://infoscience.epfl.ch/record/265758.
[22]	WANG P, CHEN P, YUAN Y, et al. Understanding convolution for semantic segmentation[EB/OL]. (2018-06-01) [2021-12-28]. https://arxiv.org/abs/1702.08502v3.
[23]	KAINZ P, URSCHLER M, SCHULTER S, et al. You should use regression to detect cells[C]//International Conference on Medical Image Computing and Computer Assisted Intervention-MICCAI. Heidelberg:Springer, 2015: 276-283.
[24]	ZHANG Y, ZHOU D, CHEN S, et al. Single-image crowd counting via multi-column convolutional neural network[C]//2016 IEEE Conf- erence on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 589-597
[25]	JIANG X, XIAO Z, ZHANG B, et al. Crowd counting and density estimation by trellis encoder-decoder networks[EB/OL]. [2021-12-28]. https://xueshu.baidu.com/usercenter/paper/show?paperid=147606t0n15c0860jb0b0010hr729346.
[26]	DAI F, LIU H, MA Y, et al. Dense scale network for crowd counting[EB/OL]. (2021-09-01) [2022-12-28]. https://dl.acm.org/doi/10.1145/3460426.3463628.
[27]	蒋妮. 基于密度估计的细胞计数方法及其在显微图像中的应用研究[D]. 杭州: 浙江大学, 2021.
	JIANG N. Research on cell counting based on density estimation and its applications in microscopic image[D]. Hangzhou: Zhejiang University, 2021 (in Chinese).