Automatic detection of pecan fruits based on Faster RCNN with FPN in orchard

Chunhua Hu, Zefeng Shi, Hailin Wei, Xiangdong Hu, Yuning Xie, Pingping Li

Abstract


Although the development of the robot picking vision system is widely applied, it is very challenging for fruit detection in orchards with complex light and environment, especially for fruit colors similar to the background. In recent, there are few studies on pecan fruit detection and location based on machine vision. In this study, an accurate and efficient pecan fruit detection method was proposed based on machine vision under natural pecan orchards. In order to solve the illumination problem, a light compensation algorithm was first utilized to process the collected samples, and then an improved Faster Region Convolutional Neural Network (Faster RCNN) with the Feature Pyramid Networks (FPN) was established to train the samples. Finally, the pecan number counting method was introduced to count the number of pecan. A total of 241 pecan images were tested, and comparison experiments were carried out. The mean average precision (mAP) of the proposed detection method was 95.932%, compared with the result without uneven illumination correction (UIC), which was increased by 0.849%, while the mAP of the Single Shot Detector (SSD)+FPN was 92.991%. In addition, the number of clusters was counted using the proposed method with an accuracy rate of 93.539% compared with the actual clusters. The results demonstrate that the proposed network has good robustness for pecan fruit detection in different illumination and various unstructured environments, and the experimental achievement has great potential for robot-picking visual systems.
Keywords: pecan fruit, fruit detection, Faster RCNN, FPN, uneven illumination correction
DOI: 10.25165/j.ijabe.20221506.7241

Citation: Hu C H, Shi Z F, Wei H L, Hu X D, Xie Y N, Li P P. Automatic detection of pecan fruits based on Faster RCNN with FPN in orchard. Int J Agric & Biol Eng, 2022; 15(6): 189–196.

Keywords


pecan fruit, fruit detection, Faster RCNN, FPN, uneven illumination correction

Full Text:

PDF

References


Kurtulmus F, Lee W S, Vardar A. Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network. Precision Agriculture, 2014; 15: 57–79.

Yang C, Lee W S, Williamson J G. Classification of blueberry fruit and leaves based on spectral signatures. Biosystems Engineering, 2012; 113(4): 351–362.

Stajnko D, Lakota, M, Hočevar M. Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Computers and Electronics in Agriculture, 2004; 42: 31–42.

Méndez Perez R, Cheein F A, Rosell-Polo J R. Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry. Computers and Electronics in Agriculture, 2017; 139: 231–242.

Tsoulias N, Paraforos D S, Xanthopoulo G, Zude-Sasse M. Apple shape detection based on geometric and radiometric features using a LiDAR laser scanner. Remote Sensing, 2020; 12(15): 2481. doi: 10.3390/rs12152481.

Zhou R, Damerow L, Sun Y, Blanke M M. Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield. Precision Agriculture, 2012; 13: 568–580.

Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.

Liakos K G, Busato P, Moshou D, Pearson S, Bochtis D. Machine learning in agriculture: A review. Sensors, 2018; 18(8): 2674. doi: 10.3390/s18082674.

Wang C L, Tang Y C, Zou X J, SiTu W M. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik, 2017; 131: 626–631.

Zhao C Y, Lee W S, He D J. Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Computers and Electronics in Agriculture, 2016; 124: 243–253. doi: 10.1016/j.ijleo.2016.11.177.

Ye Q, Huang P, Zhang Z, Zheng Y, Fu L, Yang W. Multiview learning with robust double-sided twin SVM. IEEE Transactions on Cybernetics, 2021; 52(12): 12745-12758.

Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus: IEEE, 2014; pp.580–587. doi: 10.1109/CVPR.2014.81.

Ren S, He K, Girshick, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transaction Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.

He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN, 2017. doi: 10.1109/TPAMI.2018.2844175.

Wang J M, Chen X X, Cao L, An F, Chen B Q, Xue L F, et al. Individual rubber tree segmentation based on ground-based LiDAR data and faster R-CNN of Deep Learning. Forests 2019; 10: 793. doi: 10.3390/f10090793.

Sun J, He X, Ge X, Wu X, Shen J, Song Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture 2018; 8: 196. doi: 10.3390/agriculture8120196.

Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.

Redmon J, Divvala S, Girshick R, Farhadi A. You Only Look Once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; pp.779-788. doi: 10.1109/CVPR.2016.91.

Fu C Y, Liu W, Ranga A, Tyagi A, Berg A C. DSSD: Deconvolutional Single Shot Detector, arXiv, 2017; arXiv: 1701.06659.

Kamilaris A, Prenafeta-Boldú F X. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 2018; 147: 70–90.

Yu Y, Zhang K L, Zhang D X, Yang L, Cui T. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 2019; 163: 104846. doi: 10.1016/j.compag.2019.06.001.

Vasconez J P, Delpiano J, Vougioukas S, Auat Cheein F. Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation. Computers and Electronics in Agriculture, 2020; 173: 105348. doi: 10.1016/j.compag.2020.105348.

Tian Y N, Yang G D, Wang Z, Wang H, Li E, Liang Z Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and Electronics in Agriculture, 2019; 157: 417–426.

Koirala A, Walsh K B, Wang Z, McCarthy C. Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO’. Precision Agriculture, 2019; 20: 1107–1135.

He K M, Sun J, Tang X O. Single image haze removal using dark

channel prior. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami: IEEE, 2009; pp.1956–1963. doi: 10.1109/TPAMI.2010.168.

Guo C L, Li C Y, Guo J C, Loy C C, Hou J H, Sam K W. Zero-reference deep curve estimation for low-light image enhancement. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.1777-1786. doi: 10.1109/CVPR42600.2020.00185.

Everingham M, Zisserman A, Williams C K I, Van Gool L, Allan M, Bishop C M, Chapelle O, et al. The 2005 PASCAL visual object classes challenge. In: Machine Learning Recognising Tectual Entailment, MLCW 2005, Springer, 2005; 3944: pp.117-176. doi: 10.1007/11736790_8.

He K M, Zhang X Y, Ren S Q, Sun J. Identity mappings in deep residual networks. In: Computer Vision – ECCV 2016, Springer, 2016; 9908: 630-645. doi: 10.1007/978-3-319-46493-0_38.

Fu L Y, Zhang D< Ye Q L. Recurrent thrifty attention network for remote sensing scene recognition. IEEE Transactions on Geoscience and Remote Sensing, 2021; 59(10): 8257–8268. .

Lin T Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S. Feature Pyramid Networks for Object Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017; pp.936–944. doi: 10.1109/CVPR.2017.106.

Yang Q M, Xiao D Q, Lin S C. Feeding behavior recognition for group-housed pigs with the Faster R-CNN. Computers and Electronics in Agriculture, 2018; 155: 453–460. doi: 10.1016/j.compag.2018.11.002.

Vahidi H, Klinkenberg B, Johnson B A, Moskal L M, Yan W. Mapping the individual trees in urban orchards by incorporating volunteered geographic information and very high resolution optical remotely sensed data: A template matching-based approach. Remote Sensensing, 2018; 10(7): 1134. doi: 10.3390/rs10071134.

Fan P, Lang G D, Yan B, Lei X Y, Guo P J, Liu Z J, et al. A method of segmenting apples based on gray-centered RGB color space. Remote Sensing, 2021; 13(6): 1211. doi: 10.3390/rs13061211.

Kang H W, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.

Zhuang J J, Luo S M, Hou C J, Tang Y, He Y, Xue X Y. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 2018; 152: 64–73.

Sun J, He X F, Ge X, Wu X H, Shen J F, Song Y Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture, 2018; 8(12): 196. doi: 10.3390/agriculture8120196.

Liang C X, Xiong J T, Zheng Z H, Zhong Z, Li Z H, Chen S M, et al. A visual detection method for nighttime litchi fruits and fruiting stems. Computers and Electronics in Agriculture, 2020; 169(6): 105192. doi: 10.1016/j.compag.2019.105192.

Wan Nurazwin Syazwani R, Muhammad Asraf H, Megat Syahirul Amin M A, Nur Dalila K A. Automated image identification, detection and fruit counting of top-view pineapple crown using machine learning. Alexandria Engineering Journal, 2022; 61: 1265–1276.

Chen T C, Zhang R H, Zhu L X, Zhang S A, Li X M. A method of fast segmentation for banana stalk exploited lightweight multi-feature fusion deep neural network. Machines, 2021; 9(3): 66. doi: 10.3390/ machines9030066.




Copyright (c) 2022 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office