Intelligent sorting method for assembly line based on visual positioning and model predictive control of robotic arm

Ruining Zhang, Wei Lu, Xingliang Jian, Hui Luo

Abstract


The existing steering device in the fruit and vegetable packaging assembly line cannot adjust the attitude of lettuce to a unified attitude, affecting the input and packaging process of the packaging machine. This study proposes an intelligent assembly line sorting method based on the visual positioning and model predictive control of a robotic arm. First, lightweight improvement based on the YOLOv5 is realized, the lettuce stalk in the background of the conveyor belt is promptly identified, the image of the lettuce stalk in the anchor box area is processed, and the edge contour point set is determined to extract the pixel coordinates of the optimal grasp point and mirror inclination angle of the lettuce. For the intelligent assembly line system, a robot arm kinematics model is constructed and the robot kinematics inverse solutions are calculated. Additionally, the lettuce movement speeds are dynamically measured by the vision system. A combination of the model prediction control, dynamic tracking, and rapid sorting of the lettuce by the robot claw is realized. The results show that the average detection time of a single frame image in the visual positioning part is 0.014 s, which is reduced by 50%; the accuracy and recall are 98% and 95%, respectively. The detection time is significantly reduced by ensuring accuracy. Within the current speed range of the packaging assembly line conveyor belt, the manipulator can grasp lettuce at different speeds stably and fast; the average axial error, average radial error, and adjusted average inclination angle error are 0.71 cm, 1.02 cm, and 3.79°, respectively, verifying the high efficiency and stability of the model. The proposed method of this study enables application in the intelligent sorting operation of industrial assembly lines
Keywords: YOLOv5, deep learning, image recognition, model predictive control, intelligent assembly line
DOI: 10.25165/j.ijabe.20231604.7908

Citation: Zhang R N, Lu W, Jian X L, Luo H. Intelligent sorting method for assembly line based on visual positioning and model predictive control of robotic arm. Int J Agric & Biol Eng, 2023; 16(4): 207-214.

Keywords


YOLOv5, deep learning, image recognition, model predictive control, intelligent assembly line

Full Text:

PDF

References


Kim M J, Moon Youyoun, Tou J C, Mou Beiquan, Waterland N L. Nutritional value, bioactive compounds and health benefits of lettuce (Lactuca sativa L.). Journal of Food Composition and Analysis, 2016; 49: 19–34.

Qin S. Robot tangram assembly line based on vision. Journal of Physics: Conference Series, 2022; 2229: 012017.

Ayyad A, Halwani M, Swart D, Muthusamy R, Almaskari F, Zweiri Y. Neuromorphic vision based control for the precise positioning of robotic drilling systems. Robotics and Computer-Integrated Manufacuring, 2023; 79: 102419.

Ruan D X, Zhang W T, Qian D. Feature-based autonomous target recognition and grasping of industrial robots. Personal and Ubiquitous Computing, 2023; 27: 1355-1367.

Do Q, Chang W, Chen L. Dynamic workpiece modeling with robotic pick-place based on stereo vision scanning using fast point-feature histogram algorithm. Appl Sci. 2021; 11(23): 11522. doi: 10.3390/app112311522.

Lim J, Yang P, Yuanda P, Bakkara R V, Sinaga M, Siagian H, et al. Automated pneumatic vacuum suction robotic arm with computer vision. IOP Conference Series: Materials Science and Engineering. 2020; 801(1): 012134. doi:c 10.1088/1757-899X/801/1/012134.

Janiesch C, Zschech P, Heinrich K. Machine learning and deep learning. Electron Markets, 2021; 31(3): 685–695.

Zhao Z Q, Zheng P, Xu S T, Wu X D. Object detection with deep learning: A review. IEEE Transactions on Neural Networks and Learning System, 2019; 30(11): 3212–3232.

Liu C X, Sui H G, Wang J X, Ni Z X, Ge L. Real-time ground-level building damage detection based on lightweight and accurate YOLOv5 using terrestrial images. Remote Sensing, 2022; 14(12): 2763–2763.

Zaidi S S A, Ansari M S, Aslam A, Kanwal N, Asghar M, Lee B. A survey of modern deep learning based object detection models. Digital Signal Processing, 2022; 126: 103514.

Shamshul Haq S, Lenine D, Lalitha S V N L. Performance enhancement of UPQC using Takagi–Sugeno fuzzy logic controller. International Journal of Fuzzy Systems, 2021; 23(6): 1765-1774.

Naik B B, Mehta A J. Sliding mode controller with modified sliding function for DC-DC Buck Converter. ISA Transactions, 2017; 70: 279-287.

Mannucci T, van Kampen E-J, de Visser C, Chu Q P. Safe exploration algorithms for reinforcement learning controllers. IEEE Transactions on Neural Neworks and Learning Systems, 2018; 29(4): 1069-1081.

Jahanshahi H, Sajjadi Samaneh S, Bekiros S, Aly A A. On the development of variable-order fractional hyperchaotic economic system with a nonlinear model predictive controller. Chaos, Solitons and Fractals, 2021, 144: 110698.

Pankert J, Hutter M. Perceptive model predictive control for continuous mobile manipulation. IEEE Robotics and Automation Letters, 2020; 5(4): 6177-6184.

Jia W K, Wei J M, Zhang Q, Pan N N, Niu Y, Yin X, et al. Accurate segmentation of green fruit based on optimized mask RCNN application in complex orchard. Front Plant Sci. 2022; 13: 955256. doi:10.3389/FPLS.2022.955256.

He C C, Li X B, Liu Y S, Yang B Y, Wu Z W, Tan S, et al. Combining multicolor fluorescence imaging with multispectral reflectance imaging for rapid citrus Huanglongbing detection based on lightweight convolutional neural network using a handheld device. Computers and Electronics in Agriculture, 2022; 194: 106808. doi: 10.1016/J.COMPAG.2022.106808.

de Santana Correia A, Colombini E L. Attention, please! A survey of neural attention models in deep learning. Artificial Intelligence Review, 2022; 55: 6037-6124.

Chen Z Y, Wu R H, Lin Y Y, Li C Y, Chen S Y, Yuan Z N, et al. Plant disease recognition model based on improved YOLOv5. Agronomy, 2022; 12(2): 365. doi: 10.3390/agronomy12020365.

Yan B, Fan P, Lei X Y, Liu Z J, Yang F Z. A real-time apple targets detection method for picking robot based on improved YOLOv5. Remote Sensing, 2021; 13(9): 1619. doi: 10.3390/RS13091619.

Liu Y T, Zhao J J, Luo Q Y, Shen C T, Wang R, Ding X H. Automated classification of cervical Lymph-Node-Level from ultrasound using depthwise separable convolutional swin transformer. Computers in Biology and Medicine, 2022; 148: 105821. doi: 10.1016/j.compbiomed.2022.105821.

Asghar M Z, Albogamy F R, Al-Rakhami M S, Asghar J, Rahmat M K, Alam M M, et al. Facial mask detection using depthwise separable convolutional neural network model during COVID-19 pandemic. Front Public Health, 2022; 10: 855254. doi:10.3389/FPUBH.2022.855254.

Tian Y J, Su D, Lauria S, Liu X. Recent advances on loss functions in deep learning for computer vision. Neurocomputing, 2022; 497: 129–158. doi: 10.1016/j.neucom.2022.04.127.

Yao J, Qi J M, Zhang J, Shao H M, Yang J, Li X. A real-time detection algorithm for kiwifruit defects based on YOLOv5. Electronics, 2021; 10(14): 1711. doi: 10.3390/electronics10141711.

Dong X D, Yan S, Duan C Q. A lightweight vehicles detection network model based on YOLOv5. Engineering Applications of Artificial Intelligence, 2022; 113: 104914. doi: 10.1016/j.engappai.2022.104914.

Ansari E, Akhtar M N, Abdullah M N, Othman W A F W, Bakar E A, Hawary A F, et al. Image processing of UAV imagery for river feature recognition of Kerian River, Malaysia. Sustainability, 2021; 13(17): 9568. doi: 10.3390/SU13179568.

Song Q S, Li S B, Bai Q, Yang J, Zhang X X, Li Z A, et al. Object detection method for grasping robot based on improved YOLOv5. Micromachines, 2021; 12(11): 1273. doi: 10.3390/mi12111273.

Grandia R, Farshidian F, Dosovitskiy A, Ranftl R, Hutter M. Frequency-aware model predictive control. IEEE Robotics and Automation Letters, 2019; 4(2): 1517–1524. doi: 10.1109/LRA.2019.2895882.

Pinheiro Tarcisio Carlos F, Silveira Antonio S. Constrained discrete model predictive control of an arm‐manipulator using laguerre function. Optimal Control Applications and Methods, 2020; 42(1): 160-179.

Roozbahani H, Alizadeh M, Ahomäki A, Handroos H. Coordinate-based control for a materials handling equipment utilizing real-time simulation. Autom Constr. 2021; 122: 103483. doi:10.1016/j.autcon.2020.103483.

Buizza Avanzini G, Zanchettin A M, Rocco P. Constrained model predictive control for mobile robotic manipulators. Robotica, 2018; 36(1): 19–38.




Copyright (c) 2023 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office