YOLO-RD-Apple果园异源图像遮挡果实检测模型

doi:10.11996/JG.j.2095-302X.2023030456

图学学报 ›› 2023, Vol. 44 ›› Issue (3): 456-464.DOI: 10.11996/JG.j.2095-302X.2023030456

• 图像处理与计算机视觉 • 上一篇下一篇

YOLO-RD-Apple果园异源图像遮挡果实检测模型

郝鹏飞(), 刘立群(), 顾任远

甘肃农业大学信息科学技术学院，甘肃兰州 730070

收稿日期:2022-10-25 接受日期:2023-01-15 出版日期:2023-06-30 发布日期:2023-06-30
通讯作者: 刘立群(1982-)，女，副教授，硕士。主要研究方向为智能计算与深度学习等。E-mail：llqhjy@126.com
作者简介:
郝鹏飞(1998-)，男，硕士研究生。主要研究方向为深度学习与图像处理。E-mail：hola_Cc@163.com
基金资助:
甘肃省高校教师创新基金项目(2023A-051);甘肃农业大学青年导师基金资助项目(GAU-QDFC-2020-08);甘肃省科技计划资助项目(20JR5RA032)

YOLO-RD-Apple orchard heterogenous image obscured fruit detection model

HAO Peng-fei(), LIU Li-qun(), GU Ren-yuan

College of Information Science and Technology, Gansu Agricultural University, Lanzhou Gansu 730070, China

Received:2022-10-25 Accepted:2023-01-15 Online:2023-06-30 Published:2023-06-30
Contact: LIU Li-qun (1982-), associate professor, master. Her main research interests cover intelligent computing and deep learning, etc. E-mail：llqhjy@126.com
About author:
HAO Peng-fei (1998-), master student. His main research interests cover deep learning and image processing. E-mail：hola_Cc@163.com
Supported by:
Gansu Provincial University Teacher Innovation Fund Project(2023A-051);Young Tutor Fund of Gansu Agricultural University(GAU-QDFC-2020-08);National Science Foundation of Gansu Province(20JR5RA032)

摘要/Abstract

摘要：

为探索在自然苹果园环境中对高度遮挡果实进行机器人自动化采摘，提出基于RGB与Depth图像双输入的YOLO-RD-Apple果园异源图像遮挡果实检测模型。使用轻量化的MobileNetV2并在此基础上设计的更加轻量化的MobileNetV2-Lite分别作为RGB和Depth图像的特征提取器，保证特征提取能力的同时降低网络的计算量；将CSPNet与深度可分离卷积结合SE注意力模块，提出全新的SE-DWCSP3模块对PANet结构进行改进，提升网络对于残缺苹果目标的特征提取能力；引入Soft NMS算法替代一般NMS算法，以减少对密集目标的错误抑制现象，降低被遮挡苹果的漏检率。实验结果表明，YOLO-RD-Apple在自然遮挡苹果数据集上性能优秀，在测试集上的AP值达到93.1%，较YOLOv4提升了1.4%，参数量则降低了70%，在GPU(V100)的检测速度达到40.5 FPS，较YOLOv4速度提升了12.5%，在检测精度和速度上均有不同程度地提升，同时降低了网络参数量，更加适用于实际果园苹果采摘场景。

关键词: 目标检测, 异源图像, YOLOv4, 苹果采摘, 注意力机制

Abstract:

In order to address the challenge of robotic automated picking of highly occluded fruits in natural apple orchard environments, a YOLO-RD-Apple orchard heterogenous image occlusion fruit detection model based on dual inputs of RGB and Depth images was proposed. To reduce computational effort while ensuring the feature extraction capability, the lightweight MobileNetV2 and the lighter MobileNetV2-Lite, which was designed on the basis of MobileNetV2, were utilized as feature extractors for RGB and Depth images, respectively. Combining CSPNet with depth-separable convolution to accompany the SE attention module, the new SE-DWCSP3 module was proposed to improve the PANet structure and enhance the feature extraction capability of the network for stubby apple targets. Furthermore, the Soft NMS algorithm was introduced to replace the general NMS algorithm to address the false suppression phenomenon of the algorithm for dense targets and reduce the missed detection rate of obscured apples. The experimental results demonstrated the efficacy of this model on a natural obscured apple dataset, with an AP value of 93.1% on the test set, surpassing YOLOv4 by 1.4 percentage points, a 70% reduction in the number of parameters compared to YOLOv4, and a detection speed of 40.5 FPS on GPU (V100), which is 12.5% higher than that of YOLOv4. The proposed model exhibited improved detection accuracy and speed compared with YOLOv4, while simultaneously reducing the number of network parameters, making it more applicable to actual orchard apple picking scenarios.

Key words: target detection, heterogenous images, YOLOv4, apple picking, attention mechanism

中图分类号:

TP391

郝鹏飞, 刘立群, 顾任远. YOLO-RD-Apple果园异源图像遮挡果实检测模型[J]. 图学学报, 2023, 44(3): 456-464.

HAO Peng-fei, LIU Li-qun, GU Ren-yuan. YOLO-RD-Apple orchard heterogenous image obscured fruit detection model[J]. Journal of Graphics, 2023, 44(3): 456-464.

图/表 9

图1 本文提出的YOLO-RD-Apple网络结构

Fig. 1 The YOLO-R-Apple network structure proposed in this paper

表1 本文模型骨干网络参数

Table 1 The backbone network detail parameters of the model proposed in this paper

网络层	MobileNetV2				MobileNetV2-Lite
网络层	t	c	n	s	t	c	n	s
Conv2d	-	32	1	2	-	32	1	2
Bottleneck1	1	16	1	1	1	16	1	1
Bottleneck2	6	24	2	2	6	24	1	2
Bottleneck3	6	32	3	2	6	32	1	2
Bottleneck4	6	64	4	2	6	64	1	2
Bottleneck5	6	96	3	1	6	96	1	1
Bottleneck6	6	160	3	2	6	160	1	2
Bottleneck7	6	320	1	1	6	320	1	1

图2 本文提出的SE-DWCSP模块((a) YOLOv4的叠加卷积层结构；(b)本文提出的SE-DWCSP3模块)

Fig. 2 SE-DWCSP block in this paper ((a) Stacked convolutional layer structure in YOLOv4; (b) SE-DWCSP block in this paper)

图3 训练过程损失曲线及验证集下AP曲线((a)损失曲线；(b) AP曲线)

Fig. 3 Loss curve of the training process and the curve of the validation set ((a) Loss curve; (b) AP curve)

表2 NMS和Soft-NMS下模型的检测结果(%)

Table 2 Detection results of models under NMS and Soft-NMS (%)

Model	Precision	Recall	F1	AP
Stacked convolutional layer structure	94.4	85.1	89.5	92.2
SE-DWCSP3 block	96.2	86.8	91.2	93.1

表3 NMS和Soft-NMS下模型的检测结果(%)

Table 3 Detection results of models under NMS and Soft- NMS (%)

Model	Precision	Recall	F1	AP
YOLO-R-Apple NMS	94.5	87.1	90.7	91.2
YOLO-R-Apple Soft-NMS	92.0	89.3	90.6	92.3
YOLO-RD-Apple NMS	96.1	85.0	90.2	92.8
YOLO-RD-Apple Soft-NMS	96.2	86.8	91.2	93.1

图4 本文模型在顺光和逆光环境下的检测效果((a)原图；(b) YOLO-R-Apple顺光；(c) YOLO-R-Apple逆光；(d) YOLO-RD-Apple顺光；(e) YOLO-RD-Apple逆光)

Fig. 4 Detection effect of the model under natural light and back light conditions ((a) Original images; (b) YOLO-R-Apple natural light; (c) YOLO-R-Apple back light; (d) YOLO-RD-Apple natural light; (e) YOLO-RD-Apple back light)

表4 顺光和逆光条件下模型的测试结果(%)

Table 4 Detection results of models under natural light and back light conditions (%)

Model	Natural light				Back light
Model	Precision	Recall	F1	AP	Precision	Recall	F1	AP
YOLO-R-Apple	92.4	88.9	90.6	92.5	91.6	87.3	89.4	90.4
YOLO-RD-Apple	96.8	88.4	92.4	92.9	94.2	87.8	90.9	91.7

表5 7个模型在测试集上的识别结果

Table 5 Results of the 7 models on the test set

Model	Precision (%)	Recall (%)	F1 (%)	AP (%)	Param (M)	GFLOPs (G)	FPS
YOLOv3	94.7	87.1	90.8	91.8	61.524	99.249	45.676
YOLOv4	91.8	90.0	90.9	91.7	63.938	90.816	36.018
YOLOv5-s-v6.1	95.0	87.4	91.0	93.6	7.022	10.205	54.645
YOLOv4-Tiny	93.1	85.3	89.0	91.9	5.874	10.352	154.357
Faster RCNN ResNet-50	57.0	92.7	70.6	92.0	28.275	924.058	17.599
YOLO-R-Apple	92.0	89.3	90.6	92.3	18.939	33.564	55.158
YOLO-RD-Apple	96.2	86.8	91.2	93.1	19.133	34.116	40.496

参考文献 20

[1]	田娜, 杨晓文, 单东林, 等. 我国数字农业现状与展望[J]. 中国农机化学报, 2019, 40(4): 210-213. DOI
	TIAN N, YANG X W, SHAN D L, et al. Status and prospect of digital agriculture in China[J]. Journal of Chinese Agricultural Mechanization, 2019, 40(4): 210-213. (in Chinese)
[2]	JIA W K, TIAN Y Y, LUO R, et al. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot[J]. Computers and Electronics in Agriculture, 2020, 172: 105380. DOI URL
[3]	刘晓洋, 赵德安, 贾伟宽, 等. 基于超像素特征的苹果采摘机器人果实分割方法[J]. 农业机械学报, 2019, 50(11): 15-23.
	LIU X Y, ZHAO D A, JIA W K, et al. Fruits segmentation method based on superpixel features for apple harvesting robot[J]. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50(11): 15-23. (in Chinese)
[4]	KANG H W, CHEN C. Fruit detection and segmentation for apple harvesting using visual sensor in orchards[J]. Sensors, 2019, 19(20): 4599. DOI URL
[5]	王丹丹, 宋怀波, 何东健. 苹果采摘机器人视觉系统研究进展[J]. 农业工程学报, 2017, 33(10): 59-69.
	WANG D D, SONG H B, HE D J. Research advance on vision system of apple picking robot[J]. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33(10): 59-69. (in Chinese)
[6]	FOIX S, ALENYA G, TORRAS C. Lock-in time-of-flight (ToF) cameras: a survey[J]. IEEE Sensors Journal, 2011, 11(9): 1917-1926. DOI URL
[7]	ARAD B, BALENDONCK J, BARTH R, et al. Development of a sweet pepper harvesting robot[J]. Journal of Field Robotics, 2020, 37(6): 1027-1039. DOI URL
[8]	MILELLA A, MARANI R, PETITTI A, et al. In-field high throughput grapevine phenotyping with a consumer-grade depth camera[J]. Computers and Electronics in Agriculture, 2019, 156: 293-306. DOI URL
[9]	GENÉ-MOLA J, VILAPLANA V, ROSELL-POLO J R, et al. Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities[J]. Computers and Electronics in Agriculture, 2019, 162: 689-698. DOI URL
[10]	TU S Q, XUE Y J, ZHENG C, et al. Detection of passion fruits and maturity classification using Red-Green-Blue Depth images[J]. Biosystems Engineering, 2018, 175: 156-167. DOI URL
[11]	TU S Q, PANG J, LIU H F, et al. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images[J]. Precision Agriculture, 2020, 21(5): 1072-1091. DOI
[12]	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4510-4520.
[13]	WANG C Y, LIAO H M, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2020: 1571-1580.
[14]	FERRER M, RUIZ-HIDALGO J, GREGORIO E, et al. Simultaneous fruit detection and size estimation using multitask deep neural networks[EB/OL]. [2022-08-14]. https://www.grap.udl.cat/en/publications/papple_rgb-d-size-dataset.
[15]	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 936-944.
[16]	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 8759-8768.
[17]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[18]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[19]	JOCHER G. Add SPPF() layer[EB/OL]. (2021-08-15) [2022-08-14]. https://github.com/ultralytics/yolov5/pull/4420.
[20]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID

YOLO-RD-Apple果园异源图像遮挡果实检测模型

YOLO-RD-Apple orchard heterogenous image obscured fruit detection model

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 20

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李利霞, 王鑫, 王军, 张又元 . 基于特征融合与注意力机制的无人机图像小目标检测算法 [J]. 图学学报, 2023, 44(4): 658-666.
[2]	李鑫, 普园媛, 赵征鹏, 徐丹, 钱文华 . 内容语义和风格特征匹配一致的艺术风格迁移 [J]. 图学学报, 2023, 44(4): 699-709.
[3]	邓渭铭 , 杨铁军 , 李纯纯 , 黄琳 . 基于神经网络架构搜索的铭牌目标检测方法[J]. 图学学报, 2023, 44(4): 718-727.
[4]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计 [J]. 图学学报, 2023, 44(4): 728-738.
[5]	胡欣, 周运强, 肖剑, 杨杰. 基于改进YOLOv5的螺纹钢表面缺陷检测[J]. 图学学报, 2023, 44(3): 427-437.
[6]	毛爱坤, 刘昕明, 陈文壮, 宋绍楼. 改进YOLOv5算法的变电站仪表目标检测方法[J]. 图学学报, 2023, 44(3): 448-455.
[7]	罗文宇, 傅明月. 基于YoloX-ECA模型的非法野泳野钓现场监测技术[J]. 图学学报, 2023, 44(3): 465-472.
[8]	李雨, 闫甜甜, 周东生, 魏小鹏. 基于注意力机制与深度多尺度特征融合的自然场景文本检测[J]. 图学学报, 2023, 44(3): 473-481.
[9]	王佳婧, 王晨, 朱媛媛, 王笑梅. 基于民国纸币的图元素匹配检索[J]. 图学学报, 2023, 44(3): 492-501.
[10]	吴文欢, 张淏坤. 融合空间十字注意力与通道注意力的语义分割网络[J]. 图学学报, 2023, 44(3): 531-539.
[11]	谢国波, 贺笛轩, 何宇钦, 林志毅. 基于 P-CenterNet 的光学遥感图像烟囱检测[J]. 图学学报, 2023, 44(2): 233-249.
[12]	曾伦杰 , 储珺 , 陈昭俊 . 二阶段锚框和类均衡损失的遥感图像目标检测 [J]. 图学学报, 2023, 44(2): 249-259.
[13]	熊举举 , 徐杨 , 范润泽 , 孙少聪 . 基于轻量化视觉 Transformer 的花卉识别 [J]. 图学学报, 2023, 44(2): 271-279.
[14]	陈刚 , 张培基 , 龚冬冬 , 于俊清 . 火电厂监控视频安全服检测方法研究[J]. 图学学报, 2023, 44(2): 291-297.
[15]	成浪, 敬超. 基于改进 YOLOv7 的 X 线图像旋转目标检测[J]. 图学学报, 2023, 44(2): 324-334.