YOLO-RD-Apple orchard heterogenous image obscured fruit detection model

doi:10.11996/JG.j.2095-302X.2023030456

Abstract

Abstract:

In order to address the challenge of robotic automated picking of highly occluded fruits in natural apple orchard environments, a YOLO-RD-Apple orchard heterogenous image occlusion fruit detection model based on dual inputs of RGB and Depth images was proposed. To reduce computational effort while ensuring the feature extraction capability, the lightweight MobileNetV2 and the lighter MobileNetV2-Lite, which was designed on the basis of MobileNetV2, were utilized as feature extractors for RGB and Depth images, respectively. Combining CSPNet with depth-separable convolution to accompany the SE attention module, the new SE-DWCSP3 module was proposed to improve the PANet structure and enhance the feature extraction capability of the network for stubby apple targets. Furthermore, the Soft NMS algorithm was introduced to replace the general NMS algorithm to address the false suppression phenomenon of the algorithm for dense targets and reduce the missed detection rate of obscured apples. The experimental results demonstrated the efficacy of this model on a natural obscured apple dataset, with an AP value of 93.1% on the test set, surpassing YOLOv4 by 1.4 percentage points, a 70% reduction in the number of parameters compared to YOLOv4, and a detection speed of 40.5 FPS on GPU (V100), which is 12.5% higher than that of YOLOv4. The proposed model exhibited improved detection accuracy and speed compared with YOLOv4, while simultaneously reducing the number of network parameters, making it more applicable to actual orchard apple picking scenarios.

Key words: target detection, heterogenous images, YOLOv4, apple picking, attention mechanism

CLC Number:

TP391

HAO Peng-fei, LIU Li-qun, GU Ren-yuan. YOLO-RD-Apple orchard heterogenous image obscured fruit detection model[J]. Journal of Graphics, 2023, 44(3): 456-464.

Figures/Tables 9

References 20

[1]	田娜, 杨晓文, 单东林, 等. 我国数字农业现状与展望[J]. 中国农机化学报, 2019, 40(4): 210-213. DOI
	TIAN N, YANG X W, SHAN D L, et al. Status and prospect of digital agriculture in China[J]. Journal of Chinese Agricultural Mechanization, 2019, 40(4): 210-213. (in Chinese)
[2]	JIA W K, TIAN Y Y, LUO R, et al. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot[J]. Computers and Electronics in Agriculture, 2020, 172: 105380. DOI URL
[3]	刘晓洋, 赵德安, 贾伟宽, 等. 基于超像素特征的苹果采摘机器人果实分割方法[J]. 农业机械学报, 2019, 50(11): 15-23.
	LIU X Y, ZHAO D A, JIA W K, et al. Fruits segmentation method based on superpixel features for apple harvesting robot[J]. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50(11): 15-23. (in Chinese)
[4]	KANG H W, CHEN C. Fruit detection and segmentation for apple harvesting using visual sensor in orchards[J]. Sensors, 2019, 19(20): 4599. DOI URL
[5]	王丹丹, 宋怀波, 何东健. 苹果采摘机器人视觉系统研究进展[J]. 农业工程学报, 2017, 33(10): 59-69.
	WANG D D, SONG H B, HE D J. Research advance on vision system of apple picking robot[J]. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33(10): 59-69. (in Chinese)
[6]	FOIX S, ALENYA G, TORRAS C. Lock-in time-of-flight (ToF) cameras: a survey[J]. IEEE Sensors Journal, 2011, 11(9): 1917-1926. DOI URL
[7]	ARAD B, BALENDONCK J, BARTH R, et al. Development of a sweet pepper harvesting robot[J]. Journal of Field Robotics, 2020, 37(6): 1027-1039. DOI URL
[8]	MILELLA A, MARANI R, PETITTI A, et al. In-field high throughput grapevine phenotyping with a consumer-grade depth camera[J]. Computers and Electronics in Agriculture, 2019, 156: 293-306. DOI URL
[9]	GENÉ-MOLA J, VILAPLANA V, ROSELL-POLO J R, et al. Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities[J]. Computers and Electronics in Agriculture, 2019, 162: 689-698. DOI URL
[10]	TU S Q, XUE Y J, ZHENG C, et al. Detection of passion fruits and maturity classification using Red-Green-Blue Depth images[J]. Biosystems Engineering, 2018, 175: 156-167. DOI URL
[11]	TU S Q, PANG J, LIU H F, et al. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images[J]. Precision Agriculture, 2020, 21(5): 1072-1091. DOI
[12]	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4510-4520.
[13]	WANG C Y, LIAO H M, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2020: 1571-1580.
[14]	FERRER M, RUIZ-HIDALGO J, GREGORIO E, et al. Simultaneous fruit detection and size estimation using multitask deep neural networks[EB/OL]. [2022-08-14]. https://www.grap.udl.cat/en/publications/papple_rgb-d-size-dataset.
[15]	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 936-944.
[16]	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 8759-8768.
[17]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[18]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[19]	JOCHER G. Add SPPF() layer[EB/OL]. (2021-08-15) [2022-08-14]. https://github.com/ultralytics/yolov5/pull/4420.
[20]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID

网络层	MobileNetV2				MobileNetV2-Lite
网络层	t	c	n	s	t	c	n	s
Conv2d	-	32	1	2	-	32	1	2
Bottleneck1	1	16	1	1	1	16	1	1
Bottleneck2	6	24	2	2	6	24	1	2
Bottleneck3	6	32	3	2	6	32	1	2
Bottleneck4	6	64	4	2	6	64	1	2
Bottleneck5	6	96	3	1	6	96	1	1
Bottleneck6	6	160	3	2	6	160	1	2
Bottleneck7	6	320	1	1	6	320	1	1

网络层	MobileNetV2				MobileNetV2-Lite
网络层	t	c	n	s	t	c	n	s
Conv2d	-	32	1	2	-	32	1	2
Bottleneck1	1	16	1	1	1	16	1	1
Bottleneck2	6	24	2	2	6	24	1	2
Bottleneck3	6	32	3	2	6	32	1	2
Bottleneck4	6	64	4	2	6	64	1	2
Bottleneck5	6	96	3	1	6	96	1	1
Bottleneck6	6	160	3	2	6	160	1	2
Bottleneck7	6	320	1	1	6	320	1	1

Model	Precision	Recall	F1	AP
Stacked convolutional layer structure	94.4	85.1	89.5	92.2
SE-DWCSP3 block	96.2	86.8	91.2	93.1

Model	Precision	Recall	F1	AP
Stacked convolutional layer structure	94.4	85.1	89.5	92.2
SE-DWCSP3 block	96.2	86.8	91.2	93.1

Model	Precision	Recall	F1	AP
YOLO-R-Apple NMS	94.5	87.1	90.7	91.2
YOLO-R-Apple Soft-NMS	92.0	89.3	90.6	92.3
YOLO-RD-Apple NMS	96.1	85.0	90.2	92.8
YOLO-RD-Apple Soft-NMS	96.2	86.8	91.2	93.1