Research on weed detection in vegetable seedling fields based on the improved YOLOv5 intelligent weeding robot

doi:10.11996/JG.j.2095-302X.2023020346

Abstract

Abstract:

Accurate detection of weeds is a key technology for developing automated weeding equipment. To address the problems of high detection complexity and poor robustness resulting from the complex distribution and variety of weeds, we proposed a weed detection approach for vegetable seedling based on the improved YOLOv5 algorithm and image processing, implemented on a self-developed mobile robot platform. The weed detection complexity was reduced by indirectly detecting weeds through identifying vegetables, thus improving the detection accuracy and robustness. The convolutional block attention module (CBAM) attention module was added to the backbone feature extraction network of the YOLOv5 object detection algorithm to enhance the focus of the network on vegetable targets, and the Transformer module was added to enhance the global information capture capability. The results showed that the average detection accuracy of the improved YOLOv5 algorithm for vegetable targets could reach 95.7%, which was increased by 5.8%, 6.9%, 10.3%, 13.1%, 9.0%, 5.2%, and 3.2% compared with Faster R-CNN, SSD, EfficientDet, RetinaNet, YOLOv3, YOLOv4, and YOLOv5, respectively. The average detection time of the algorithm for a single run was 11 ms, indicating good real-time performance. The method defined green plants outside the vegetable border as weeds, and combined the extreme green (ExG) with the OTSU threshold segmentation method to segment weeds from the soil background. Finally, the weed connectivity domain was marked, followed by outputting the weed plasmids and detection frames. The proposed method could provide a technical reference for automated precision weeding in agriculture.

Key words: weeding robot, weed detection, vegetable identification, YOLOv5, attention mechanism

CLC Number:

TP391

ZHANG Wei-kang, SUN Hao, CHEN Xin-kai, LI Xu-bing, YAO Li-gang, DONG Hui. Research on weed detection in vegetable seedling fields based on the improved YOLOv5 intelligent weeding robot[J]. Journal of Graphics, 2023, 44(2): 346-356.

Figures/Tables 19

Fig. 1 Weeding robot

Fig. 2 Structure diagram of robot system

Fig. 3 Robot working process chart

Fig. 4 Weed detection process

Fig. 5 YOLOv5-CBTR algorithm network structure

Fig. 6 CBAM attention network structure

Fig. 7 Transformer encoder structure

Fig. 8 Weed detection schematic

Fig. 9 Data acquisition environment ((a) Greenhouse; (b) Vegetable growing conditions)

Fig. 10 Crop and weed distribution ((a) Weed grow with crop; (b) Dense weed distribution; (c) Sparse weed distribution; (d) Weed away from crop)

Fig. 11 Data augmentation example ((a) Original; (b) Flip; (c) Colour enhancement; (d) Gaussian noise)

Table 1 Experiment platform configuration

配置名称	版本参数
操作系统	Ubuntu18.04
CPU	Intel(R)Core(TM)i9-10920X
内存	32 G
GPU	NVIDIA GeForce RTX3070
深度学习框架	Pytorch1.8.0

Fig. 12 Model training curve ((a) Loss function curve; (b) PR curve)

Fig. 13 mAP curve comparison

Table 2 Comparison results of performance indicators of different algorithms

Model	P (%)	R (%)	mAP (%)	参数量(MB)	时间(ms·fps^-1)
Faster R-CNN	92.5	79.4	89.9	113.6	73
SSD	92.1	80.9	88.8	97.1	35
EfficientDet	93.6	77.4	85.4	20.8	44
RetinaNet	93.5	74.8	82.6	146.0	52
YOLOv3	90.4	78.9	86.7	246.5	38
YOLOv4	91.4	83.3	90.5	256.3	33
YOLOv5	90.3	87.1	92.5	14.4	9
YOLOv5-CBTR	94.5	93.1	95.7	16.1	11

Table 3 Ablation experimental results

Model	CBAM	Transformer	P (%)	R (%)	mAP (%)
YOLOv5	-	-	90.3	87.1	92.5
	√	-	93.8	91.3	94.2
	-	√	92.6	90.7	93.8
	√	√	94.5	93.1	95.7

Fig. 14 Comparison of test results ((a) Original; (b) Faster R-CNN; (c) SSD; (d) YOLOv5; (e) YOLOv5-CBTR)

Fig. 15 Weed detection results ((a) Vegetable detection results; (b) Weed segmentation results)

Fig. 16 Weeding experiment

References 28

[1]	KHAN S, TUFAIL M, KHAN M T, et al. Deep learning-based identification system of weeds and crops in strawberry and pea fields for a precision agriculture sprayer[J]. Precision Agriculture, 2021, 22(6): 1711-1727. DOI
[2]	姜红花, 张传银, 张昭, 等. 基于Mask R-CNN的玉米田间杂草检测方法[J]. 农业机械学报, 2020, 51(6): 220-228, 247.
	JIANG H H, ZHANG C Y, ZHANG Z, et al. Detection method of corn weed based on mask R-CNN[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(6): 220-228, 247. (in Chinese)
[3]	COLEMAN G, SALTER W, WALSH M. OpenWeedLocator (OWL): an open-source, low-cost device for fallow weed detection[J]. Scientific Reports, 2022, 12: 170. DOI PMID
[4]	MCCOOL C, BEATTIE J, FIRN J, et al. Efficacy of mechanical weeding tools: a study into alternative weed management strategies enabled by robotics[J]. IEEE Robotics and Automation Letters, 2018, 3(2): 1184-1190.
[5]	LI Y, GUO Z Q, SHUANG F, et al. Key technologies of machine vision for weeding robots: a review and benchmark[J]. Computers and Electronics in Agriculture, 2022, 196: 106880. DOI URL
[6]	孟庆宽, 张漫, 杨晓霞, 等. 基于轻量卷积结合特征信息融合的玉米幼苗与杂草识别[J]. 农业机械学报, 2020, 51(12): 238-245, 303.
	MENG Q K, ZHANG M, YANG X X, et al. Recognition of maize seedling and weed based on light weight convolution and feature fusion[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(12): 238-245, 303. (in Chinese)
[7]	NGUYEN THANH LE V, APOPEI B, ALAMEH K. Effective plant discrimination based on the combination of local binary pattern operators and multiclass support vector machine methods[J]. Information Processing in Agriculture, 2019, 6(1): 116-131. DOI URL
[8]	CHEN Y J, WU Z N, ZHAO B, et al. Weed and corn seedling detection in field based on multi feature fusion and support vector machine[J]. Sensors: Basel, Switzerland, 2020, 21(1): 212.
[9]	温德圣, 许燕, 周建平, 等. 自然光照影响下基于深度卷积神经网络和颜色迁移的杂草识别方法[J]. 中国科技论文, 2020, 15(3): 287-292.
	WEN D S, XU Y, ZHOU J P, et al. Weed identification method based on deep convolutional neural network and color migration under the influence of natural illumination[J]. China Sciencepaper, 2020, 15(3): 287-292. (in Chinese)
[10]	TOO E C, LI Y J, NJUKI S, et al. A comparative study of fine-tuning deep learning models for plant disease identification[J]. Computers and Electronics in Agriculture, 2019, 161: 272-279. DOI URL
[11]	GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 580-587.
[12]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. DOI PMID
[13]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[M]//Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
[14]	TAN M X, PANG R M, LE Q V. EfficientDet: scalable and efficient object detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10778-10787.
[15]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2999-3007.
[16]	REDMON J, FARHADI A. YOLOv3: an incremental improvement[EB/OL]. (2018-04-08) [2022-05-02]. https://arxiv.org/abs/1804.02767.
[17]	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. (2020-04-23) [2022-05-02]. https://arxiv.org/abs/2004.10934.
[18]	YING B Y, XU Y C, ZHANG S, et al. Weed detection in images of carrot fields based on improved YOLOv4[J]. Traitement Du Signal, 2021, 38(2): 341-348. DOI URL
[19]	樊湘鹏, 周建平, 许燕, 等. 基于优化Faster R-CNN的棉花苗期杂草识别与定位[J]. 农业机械学报, 2021, 52(5): 26-34.
	FAN X P, ZHOU J P, XU Y, et al. Identification and localization of weeds based on optimized faster R-CNN in cotton seedling stage[J]. Transactions of the Chinese Society for Agricultural Machinery, 2021, 52(5): 26-34. (in Chinese)
[20]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 779-788.
[21]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID
[22]	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2117-2125.
[23]	REZATOFIGHI H, TSOI N, GWAK J Y, et al. Generalized intersection over union: a metric and a loss for bounding box regression[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 658-666.
[24]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]// Computer Vision-ECCV2018. Cham: Springer International Publishing, 2018: 3-19.
[25]	奉志强, 谢志军, 包正伟, 等. 基于改进YOLOv5的无人机实时密集小目标检测算法[J/OL]. 航空学报, 2022: 1-15. [2022- 05-11]. https://kns.cnki.net/kcms/detail/11.1929.V.20220509.2316.010.html.
	FENG Z Q, XIE Z J, BAO Z W, et al. Real-time dense small object detection algorithm for UAV based on improved YOLOv5[J/OL]. Acta Aeronautica et Astronautica Sinica, 2022: 1-15. [2022-05-11]. . (in Chinese)
[26]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. [2022-05-02]. https://arxiv.org/abs/2010.11929.
[27]	吴兰兰, 熊利荣, 彭辉. 基于RGB植被指数的大田油菜图像分割定量评价[J]. 华中农业大学学报, 2019, 38(2): 109-113.
	WU L L, XIONG L R, PENG H. Quantitative evaluation of in-field rapeseed image segmentation based on RGB vegetation indices[J]. Journal of Huazhong Agricultural University, 2019, 38(2): 109-113. (in Chinese)
[28]	ELSTONE L, HOW K Y, BRODIE S, et al. High speed crop and weed identification in lettuce fields for precision weeding[J]. Sensors: Basel, Switzerland, 2020, 20(2): 455.