Multi-class detection of cherry tomatoes using improved Yolov4-tiny model

Fu Zhang, Zijun Chen, Shaukat Ali, Ning Yang, Sanling Fu, Yakun Zhang

Abstract


The rapid and accurate detection of cherry tomatoes is of great significance to realizing automatic picking by robots. However, so far, cherry tomatoes are detected as only one class for picking. Fruits occluded by branches or leaves are detected as pickable objects, which may cause damage to the plant or robot end-effector during picking. This study proposed the Feature Enhancement Network Block (FENB) based on YOLOv4-Tiny to solve the above problem. Firstly, according to the distribution characteristics and picking strategies of cherry tomatoes, cherry tomatoes were divided into four classes in the nighttime, and daytime included not occluded, occluded by branches, occluded by fruits, and occluded by leaves. Secondly, the CSPNet structure with the hybrid attention mechanism was used to design the FENB, which pays more attention to the effective features of different classes of cherry tomatoes while retaining the original features. Finally, the Feature Enhancement Network (FEN) was constructed based on the FENB to enhance the feature extraction ability and improve the detection accuracy of YOLOv4-Tiny. The experimental results show that under the confidence of 0.5, average precision (AP) of non-occluded, branch-occluded, fruit-occluded, and leaf-occluded fruit over the day test images were 95.86%, 92.59%, 89.66%, and 84.99%, respectively, which were 98.43%, 95.62%, 95.50%, and 89.33% on the night test images, respectively. The mean Average Precision (mAP) of four classes over the night test set was higher (94.72%) than that of the day (90.78%), which were both better than YOLOv4 and YOLOv4-Tiny. It cost 32.22 ms to process a 416×416 image on the GPU. The model size was 39.34 MB. Therefore, the proposed model can provide a practical and feasible method for the multi-class detection of cherry tomatoes.
Keywords: cherry tomatoes, deep learning, data augmentation, YOLOv4, occlusion, multi-class detection
DOI: 10.25165/j.ijabe.20231602.7744

Citation: Zhang F, Chen Z J, Ali S, Yang N, Fu S L, Zhang Y K. Multi-class detection of cherry tomatoes using improved YOLOv4-Tiny. Int J Agric & Biol Eng, 2023; 16(2): 225-231.

Keywords


y tomatoes, deep learning, data augmentation, YOLOv4, occlusion, multi-class detection

Full Text:

PDF

References


Rong J C, Wang P B, Yang Q, Huang F. A field-tested harvesting robot for oyster mushroom in greenhouse. Agronomy, 2021; 11(6): 1210. doi: 10.3390/agronomy11061210.

Zhang F, Chen Z J, Wang Y F, Bao R F, Chen X G, Fu S L, et al. Research on flexible end-effectors with humanoid grasp function for small spherical fruit picking. Agriculture, 2023; 13(1): 123. doi: 10.3390/agriculture13010123.

Rong J C, Wang P B, Wang T J, Ling H, Yuan T. Fruit pose recognition and directional orderly grasping strategies for tomato harvesting robots. Computers and Electronics in Agriculture, 2022; 202: 107430. doi: 10.1016/j.compag.2022.107430.

Afroza A, Ambreen N, Baseerat A; Nigeena N, Ahmad S P, Azrah I S, Amreena S; Insha J, Majid R. Evaluation of Cherry Tomato (Solanum lycopersicum L. var. cerasiforme) Genotypes for Yield and Quality Traits. Journal of Community Mobilization and Sustainable Development, 2021; 16(1): 72-76.

Yamamoto K, Guo W, Ninomiya S S. Node detection and internode length estimation of tomato seedlings based on image analysis and machine learning. Sensors, 2016; 16(7): 1044.

Wang Z L. Underwood J. Walsh KB. Machine vision assessment of mango orchard flowering. Computers and Electronics in Agriculture, 2018; 151: 501–511.

Wu J G, Zhang B H, Zhou J, Xiong Y J, Gu B X, Yang X L. Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots. Sensors, 2019; 19(3): 612- 612.

Xiong J T, Lin R, Liu Z, He Z L, Tang L Y, Yang Z G Zou X J. The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 2018; 166: 44-57.

Bechar A, Vigneault C. Agricultural robots for field operations: Concepts and components. Biosystems Engineering, 2016; 149: 94-111.

Silwal A, Davidson J R, Karkee M, Mo C K, Zhang Q, Lewis K. Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 2017; 34(6: 1140-1159.

Silwal A, Karkee M, Zhang Q. A hierarchical approach to apple identification for robotic harvesting. Transactions of the ASABE, 2016; 59(5): 1079-1086.

Guo Y M, Liu Y, Oerlemans A, Lao S Y, Lew M S. Deep learning for visual understanding: A review. Neurocomputing, 2016; 187: 27-48.

He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 770-778.

Liu F, Liu Y K, Lin S, Guo W Z, Xu F, Zhang B. Fast recognition method for tomatoes under complex environments based on improved YOLO. Transactions of the Chinese Society for Agriculture Machinery, 2020; 51(6): 229-237. (in Chinese)

Vougioukas S G. Annual Review of Control, Robotics, and Autonomous Systems. Annual Reviews, 2019; 2: 365-392.

Zhang F, Chen Z J, Bao R F, Zhang C C, Wang Z H. Recognition of dense cherry tomatoes based on improved YOLOv4-LITE lightweight neural network. Transactions of the Chinese Society of Agricultural Engineering, 2021; 37(16): 270-278. (in Chinese)

Xu Z F, Jia R S, Liu Y B, Zhao C Y, Sun H M. Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots. IEEE Access, 2020; 8: 55289 - 55299.

Zhang W J, Zhao X X, Ding R R, Zhang Z, Jiang H H, Liu P Z. A Detection and Recognition Method for Tomato on Faster R-CNN Algorithm. Journal of Shandong Agricultural University (Natural Science Edition), 2021; 52(4): 624-630. (in Chinese)

Xu C, Xiong Z, Jiang X P, Deng M, Huang G C. Design and research of the cluster tomato picking robot. Modern Agricultural Equipment, 2021; 42(6): 15-23. (in Chinese)

Zhang Q, Liu F P, Jiang X P, Xiong Z, Xu C. Motion planning method and experiments of tomato bunch harvesting manipulator. Transactions of the CSAE, 2021; 39(7): 149-156. (in Chinese)

Gao F F, Fu L S, Zhang X, Majeed Y, Li R, Karkee M, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster RCNN. Computers and Electronics in Agriculture, 2020; 176: 105634. doi: 10.1016/j.compag.2020.105634.

Suo R, Gao F F, Zhou Z X, Fu L X, Song Z Z, Dhupia J, et al. Improved multi-classes kiwifruit detection in orchard to avoid collisions during robotic picking. Computers and Electronics in Agriculture, 2021; 182: 106052. doi: 10.1016/j.compag.2021.106052.

Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137-1149.

Zhao D A, Wu R D, Liu X Y, Zhao Y Y. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background. Transactions of the CSAE, 2019; 35(3): 164-173. (in Chinese)

Wang C, Bochkovskiy A, Liao H M. Scaled-YOLOv4: Scaling cross stage partial network. In: 20221 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021; pp.13024-13033. doi: 10.1109/CVPR46437.2021.01283.

Li H P, Li C Y, Li G B, Chen L X. A real-time table grape detection method based on improved YOLOv4-tiny network in complex background. Biosystems Engineering, 2021; 212: 347-359.

Xu B, Wang N Y, Chen T Q, Li M. Empirical evaluation of rectified activations in convolutional network. arXiv preprint, 2015; arXiv:1505.00853.

Zheng Z H, Wang P, Liu W, Li J Z, Ye R G, Ren D W. Distance-IoU loss: Faster and better learning for bounding box regression. arXiv preprint, 2020; arXiv:1911.08287.

Wang C Y, Liao H M, Wu Y H, Chen P Y, Hsieh J W, Yeh I H. CSPNet: A new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020; pp.1571-1580.

Woo S, Park J C, Lee J, Lweon I. CBAM: Convolutional Block Attention Module. In: Computer Vision - ECCV, 2018; pp.3-19.




Copyright (c) 2023 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office