Fusion of the deep networks for rapid detection of branch-infected aeroponically cultivated mulberries using multimodal traits

Osama Elsherbiny, Jianmin Gao, Yinan Guo, Mazhar H. Tunio, Abdallah H. Mosha

Abstract


Automatic diagnosis of diseases in aeroponically cultivated branches is crucial for enhancing the efficacy of root development and overall plant survivability during propagation. Deep learning and visible imaging offer potential for precise health assessment, despite challenges in feature selection and model design, impacting diagnostic accuracy and effectiveness. The primary objective of this study is to explore a hybrid deep network that integrates multimodal data, such as texture and color attributes, as well as image color modes, to accurately detect the presence of mildew on mulberry branches. The proposed framework incorporates a Convolutional Neural Network (CNN) and Gated Recurrent Units (GRU). Various color modes were utilized, including grayscale, RGB (Red-Green-Blue), HSV (Hue-Saturation-Value), and CMYK (Cyan-Magenta-Yellow-Black). The traits based on RGB consist of nineteen vegetation color indices (VIs) and six texture variables obtained from the gray-level co-occurrence matrix (GLCM). The outcomes demonstrated that the CNNCMYK-GRUf network effectively integrates CMYK image data and color-texture features for tracking mulberry branch health during aeroponic propagation. It achieved a validation accuracy (Ac) of 99.50%, with classification precision (Pr), recall (Re), and F-measure (Fm) at the same level. Additionally, it obtained an intersection over union (IoU) of 98.90% and a loss value of 0.034. This network exhibited superior performance compared to the model that relied solely on individual image attributes, surpassing other deep networks such as Vision Transformers (Ac=94.80%), Swin Transformers (Ac=89.80%), and Multi-Layer Perceptrons (Ac=88.30%). Thus, the proposed methodology is capable of precisely assessing the health of mulberry shoots, enabling the swift deployment of intelligent aeroponic systems. Furthermore, adapting the developed model for mobile platforms could enhance its accessibility and promote sustainable, efficient agricultural practices.
Key words: mulberry twigs health; digital imagery; VIs-GLCM; top features selection; CNN-GRU; aeroponic system
DOI: 10.25165/j.ijabe.20251802.8666

Citation: Elsherbiny O, Gao J M, Guo Y N, Tunio M H, Mosha A H. Fusion of the deep networks for rapid detection of branch-infected
aeroponically cultivated mulberries using multimodal traits. Int J Agric & Biol Eng, 2025; 18(2): 75–88.

Keywords


mulberry twigs health; digital imagery; VIs-GLCM; top features selection; CNN-GRU; aeroponic system

Full Text:

PDF

References


Duragkar H A. Identifying diseases in mulberry leaves that affects silk production: A deep learning approach. PhD dissertation. Dublin: National College of Ireland, 2023; 17p.

Food and Agriculture Organization of the United Nation (FAO). International plant protection convention. Plant Health and Food Security. 2017.

Kim K H, Kabir E, Jahan S A. Exposure to pesticides and the associated human health effects. Science of the Total Environment, 2017; 575: 525–535.

Knillmann S, Liess M. Pesticide effects on stream ecosystems. In: Schröter M, Bonn A, Klotz S, Seppelt R, Baessler C, editors. Atlas of ecosystem services: Drivers, risks, and societal responses. Springer. 2019; 211–214.

Sofo A, Zanella A, Ponge J F. Soil quality and fertility in sustainable agriculture, with a contribution to the biological classification of agricultural soils. Soil Use and Management, 2022; 38(2): 1085–1112.

Lakhiar I A, Gao J, Syed T N, Chandio F A, Buttar N A. Modern plant cultivation technologies in agriculture under controlled environment: A review on aeroponics. Journal of Plant Interactions, 2018; 13(1): 338–352.

Luo Y H. Research and experiment on key technologies of root zone environment control in aeroponics cultivation system. PhD dissertation. Changsha: Hunan Agricultural University, 2020; 170p. (in Chinese)

Kumar S P, Sampath B, Kumar R S, Babu B H, Ahalya N. Hydroponics, aeroponics, and aquaponics technologies in modern agricultural cultivation. In: Trends, Paradigms, and Advances in Mechatronics Engineering. 2023; pp.223–241. doi: 10.4018/978-1-6684-5887-7.ch012.

Tokunaga H, Anh N H, Dong N V, Ham L H, Hanh N T, Hung N, et al. An efficient method of propagating cassava plants using aeroponic culture. Journal of Crop Improvement, 2020; 34(1): 64–83.

Yu Y Y, Jia D D, Zhuang Z, Zhu C X, Liu A Q. Research progress on aeroponics for plants. Jiangsu Journal of Agricultural Sciences, 2019; 47(18): 38–42. (in Chinese)

Tunio M H, Gao J, Shaikh S A, Lakhiar I A, Qureshi W A, Solangi K A, et al. Potato production in aeroponics: An emerging food growing system in sustainable agriculture for food security. Chilean Journal of Agricultural Research, 2020; 80(1): 118–132.

Fasciolo B, Awouda A, Bruno G, Lombardi F. A smart aeroponic system for sustainable indoor farming. Procedia CIRP, 2023; 116: 636–641.

Sadek N, kamal N, Shehata D. Internet of Things based smart automated indoor hydroponics and aeroponics greenhouse in Egypt. Ain Shams Engineering Journal, 2024; 15(2): 102341.

Ghimire A, Dahal M, Karki R. Hydroponics: An innovative approach to urban agriculture. Nepalese Journal of Agricultural Sciences, 2023; 25: 89–98.

Fernando S D, Gamage A, De Silva D H. Machine learning to aid in the process of disease detection and management in soilless farming. In: 2022 IEEE 7th International conference for Convergence in Technology (I2CT), Mumbai, India: IEEE. 2022; pp.1–5. doi: 10.1109/I2CT54291.2022.9824206.

Abbaspour-Gilandeh Y, Molaee A, Sabzi S, Nabipur N, Shamshirband S, Mosavi A. A combined method of image processing and artificial neural network for the identification of 13 Iranian rice cultivars. Agronomy, 2020; 10(1): 117.

Elsherbiny O, Elaraby A, Alahmadi M, Hamdan M, Gao J. Rapid grapevine health diagnosis based on digital imaging and deep learning. Plants, 2024; 13(1): 135.

Prey L, von Bloh M, Schmidhalter U. Evaluating RGB imaging and multispectral active and hyperspectral passive sensing for assessing early plant vigor in winter wheat. Sensors, 2018; 18: 2931.

Elsherbiny O, Gao J, Ma M, Guo Y, Tunio M H, Mosha A H. Advancing lettuce physiological state recognition in IoT aeroponic systems: A meta-learning-driven data fusion approach. European Journal of Agronomy, 2024; 161: 127387.

Li L, Zhang Q, Huang D F. A review of imaging techniques for plant phenotyping. Sensors, 2014; 14: 20078–20111.

Elsherbiny O, Zhou L, He Y, Qiu Z J. A novel hybrid deep network for diagnosing water status in wheat crop using IoT-based multimodal data. Computers and Electronics in Agriculture, 2022; 203: 107453.

Liu Y, Feng H, Yue J, Li Z, Yang G, Song X, et al. Remote-sensing estimation of potato above-ground biomass based on spectral and spatial features extracted from high-definition digital camera images. Computers and Electronics in Agriculture, 2022; 198: 107089.

Zhou L, Zhang C, Taha M F, Qiu Z J, He Y. Determination of leaf water content with a portable NIRS system based on deep learning and information fusion analysis. Transactions of the ASABE, 2021; 64(1): 127–135.

Li W, Sun Z, Lu S, Omasa K. Estimation of the leaf chlorophyll content using multiangular spectral reflectance factor. Plant, Cell & Environment, 2019; 42(11): 3152–3165.

Taha M F, Abdalla A, ElMasry G, Gouda M, Zhou L, Zhao N, et al. Using deep convolutional neural network for image-based diagnosis of nutrient deficiencies in plants grown in aquaponics. Chemosensors, 2022; 10(2): 45.

Zhou L, Xiao Q, Taha M F, Xu C J, Zhang C. Phenotypic analysis of diseased plant leaves using supervised and weakly supervised deep learning. Plant Phenomics, 2023; 5: 0022.

Ruwanpathirana P P, Sakai K, Jayasinghe G Y, Nakandakari T, Yuge K, Wijekoon W M J C, et al. Evaluation of sugarcane crop growth monitoring using vegetation indices derived from RGB-based UAV images and machine learning models. Agronomy, 2024; 14(9): 2059.

Mathew A, Antony A, Mahadeshwar Y, Khan T, Kulkarni A. Plant disease detection using GLCM feature extractor and voting classification approach. Materials Today: Proceedings, 2022; 58: 407–415.

Bhimte N R, Thool V R. Diseases detection of cotton leaf spot using image processing and SVM classifier. In: 2018 Second international conference on intelligent computing and control systems (ICICCS), Madurai, India: IEEE, 2018; pp.340–344.

Alomar K, Aysel H I, Cai X. Data augmentation in classification and segmentation: A survey and new strategies. Journal of Imaging, 2023; 9(2): 46.

Jin X, Jie L, Wang S, Qi H J, Li S W. Classifying wheat hyperspectral pixels of healthy heads and Fusarium head blight disease using a deep neural network in the wild field. Remote Sensing, 2018; 10(3): 395.

Ullah Z, Alsubaie N, Jamjoom M, Alajmani S H, Saleem F. EffiMob-Net: A deep learning-based hybrid model for detection and identification of tomato diseases using leaf images. Agriculture, 2023; 13(3): 737.

Nery E W, Kubota L T. Sensing approaches on paper-based devices: A review. Analytical and Bioanalytical Chemistry, 2013; 405: 7573–7595.

Yossy E H, Pranata J, Wijaya T, Hermawan H, Budiharto W. Mango fruit sortation system using neural network and computer vision. Procedia Computer Science, 2017; 116: 596–603.

Feng W, Shen W Y, He L, Duan J Z, Guo B B, Li Y X, et al. Improved remote sensing detection of wheat powdery mildew using dual-green vegetation indices. Precision Agriculture, 2016; 17: 608–627.

Zhu W J, Feng Z K, Dai S Y, Zhang P P, Wei X H. Using UAV multispectral remote sensing with appropriate spatial resolution and machine learning to monitor wheat scab. Agriculture, 2022; 12(11): doi: 10.3390/agriculture12111785.

Peng Y, Zhao S Y, Liu J Z. Fused-deep-features based grape leaf disease diagnosis. Agronomy, 2021; 11(11): 2234.

Netto A F A, Martins R N, de Souza G S, de Moura Araújo G, de Almeida S L H, et al. Segmentation of RGB images using different vegetation indices and thresholding methods. Nativa, 2018; 6(4): 389–394.

Goddijn-Murphy L, Dailloux D, White M, Bowers D. Fundamentals of in situ digital camera methodology for water quality monitoring of coast and ocean. Sensors, 2009; 9(7): 5825–5843.

Sellaro R, Crepy M, Trupkin S A, Karayekov E, Buchovsky A S, Rossi C, et al. Cryptochrome as a sensor of the blue/green ratio of natural radiation in Arabidopsis. Plant Physiology, 2010; 154(1): 401–409.

Verrelst J, Schaepman M E, Koetz B, Kneubühler M. Angular sensitivity analysis of vegetation indices derived from CHRIS/PROBA data. Remote Sensing of Environment, 2008; 112(5): 2341–2353.

Woebbecke D M, Meyer G E, Von Bargen K, Mortensen D A. Color indices for weed identification under various soil, residue, and lighting conditions. Transactions of the ASAE, 1995; 38(1): 259–269.

Kawashima S, Nakatani M. An algorithm for estimating chlorophyll content in leaves using a video camera. Annals of Botany, 1998; 81(1): 49–54.

Woebbecke D M, Meyer G E, Von Bargen K, Mortensen D A. Plant species identification, size, and enumeration using machine vision techniques on near-binary images. In: Optics in Agriculture and Forestry, International Society for Optics and Photonics, 1993; pp.208–219.

Mao W H, Wang Y M, Wang Y Q. Real-time detection of between-row weeds using machine vision. In: 2003 ASAE Annual Meeting, American Society of Agricultural and Biological Engineers, 2003. doi: 10.13031/2013.15381.

Gerardo R, de Lima I P. Applying RGB-Based vegetation indices obtained from uas imagery for monitoring the rice crop at the field scale: A case study in Portugal. Agriculture, 2023; 13(10): 1916.

Hague T, Tillett N D, Wheeler H. Automated crop and weed monitoring in widely spaced cereals. Precision Agriculture, 2006; 7: 21–32.

Saberioon M M, Amin M S, Anuar A R, Gholizadeh A, Wayayok A, Khairunniza-Bejo S. Assessment of rice leaf chlorophyll content using visible bands at different growth stages at both the leaf and canopy scale. International Journal of Applied Earth Observation and Geoinformation, 2014; 32: 35–45.

Kataoka T, Kaneko T, Okamoto H, Hata S. Crop growth estimation system using machine vision. In: Proceedings 2003 IEEE/ASME international conference on advanced intelligent mechatronics (AIM 2003), Kobe, Japan: IEEE, 2003; pp.b1079–b1083.

Guijarro M, Pajares G, Riomoros I, Herrera P J, Burgos-Artizzu X P, Ribeiro A. Automatic segmentation of relevant textures in agricultural images. Computers and Electronics in Agriculture, 2011; 75(1): 75–83.

Hall-Beyer M. GLCM Texture: A Tutorial v. 1.0 through 2.7. Available: http://hdl.handle.net/1880/51900. Accessed on[2023-04-04].

Latif M A, Afshan N, Mushtaq Z, Khan N A, Irfan M, Nowakowski G. Enhanced classification of coffee leaf biotic stress by synergizing feature concatenation and dimensionality reduction. IEEE Access, 2023; 100887–100906. doi: 10.1109/ACCESS.2023.3314590.

Maheswari M U, Ramani R A. Comparative study of agricultural crop yield prediction using machine learning techniques. In: 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS), 2023; pp.1428–1433. doi: 10.1109/ICACCS57279.2023.10112854.

Kamilaris A, Prenafeta-Boldú F X. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 2018; 147: 70–90.

Ubbens J R, Stavness I. Deep plant phenomics: a deep learning platform for complex plant phenotyping tasks. Frontiers in Plant Science, 2017; 8: 1190.

Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015; pp.448–456.

Bi L, Wally O, Hu G, Tenuta A U, Kandel Y R, Mueller D S. A transformer-based approach for early prediction of soybean yield using time-series images. Frontiers in Plant Science, 2023; 14: 1173036.

Chung J, Gulcehre C, Cho K, Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint, 2014; In press. doi: 10.48550/arXiv.1412.3555.

Gaagai A, Aouissi H A, Bencedira S, Hinge G, Athamena A, Heddam S, et al. Application of water quality indices, machine learning approaches, and GIS to identify groundwater quality for irrigation purposes: A case study of Sahara Aquifer, Doucen Plain, Algeria. Water, 2023; 15(2): 289.

Barbedo J G A. Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification. Computers and Electronics in Agriculture, 2018; 153: 46–53.

Iwana B K, Uchida S. An empirical survey of data augmentation for time series classification with neural networks. Plos one, 2021; 16(7): e0254841.

Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv: 2010.11929, 2020; In press. doi: 10.48550/arXiv.2010.11929.

Liu Z, Lin Y T, Cao Y, Hu H, Wei Y X, Zhang Z, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada: IEEE, 2021; pp.10012–10022.

Tolstikhin I O, Houlsby N, Kolesnikov A, Beyer L, Zhai X, Unterthiner T, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 2021; 34: 24261–24272.

Summaira J, Li X, Shoib A M, Li S, Abdul J. Recent advances and trends in multimodal deep learning: A review. arXiv preprint arXiv: 2105.11087, 2021; In press. doi: 10.48550/arXiv.2105.11087.

Zhong F M, Chen Z K, Zhang Y C, Xia F. Zero-and few-shot learning for diseases recognition of Citrus aurantium L. using conditional adversarial autoencoders. Computers and Electronics in Agriculture, 2020; 179: 105828.

Zhu W J, Li J Y, Li L, Wang A C, Wei X H, Mao H P. Nondestructive diagnostics of soluble sugar, total nitrogen and their ratio of tomato leaves in greenhouse by polarized spectra–hyperspectral data fusion. Int J Agric & Biol Eng, 2020; 13(2): 189–197.

Wakamori K, Mizuno R, Nakanishi G, Mineno H. Multimodal neural network with clustering-based drop for estimating plant water stress. Computers and Electronics in Agriculture, 2020; 168: 105118.

Elmetwalli A H, Mazrou Y S A, Tyler A N, Hunter P D, Elsherbiny O, Yaseen Z M, et al. Assessing the efficiency of remote sensing and machine learning algorithms to quantify wheat characteristics in the Nile Delta Region of Egypt. Agriculture, 2022, 25; 12(3): 332. doi: 10.3390/agriculture12030332.

Narimani M, Hajiahmad A, Moghimi A, Alimardani R, Rafiee S, Mirzabe A H. Developing an aeroponic smart experimental greenhouse for controlling irrigation and plant disease detection using deep learning and IoT. In: 2021 ASABE Annual International Virtual Meeting, Michigan: American Society of Agricultural and Biological Engineers, 2021. doi:10.13031/aim.202101252.

Kurup R V, Anupama M A, Vinayakumar R, Sowmya V, Soman K P. Capsule network for plant disease and plant species classification. In Computational Vision and Bio-Inspired Computing (ICCVBIC 2019), Springer, 2020; pp.413–421. doi: 10.1007/978-3-030-37218-7_47.

Karlekar A, Seal A. SoyNet: Soybean leaf diseases classification. Computers and Electronics in Agriculture, 2020; 172: 105342.

Taha M F, Mao H, Mousa S, Zhou L, Wang Y, Elmasry G, et al. Deep learning-enabled dynamic model for nutrient status detection of aquaponically grown plants. Agronomy, 2024; 14: 2290.

Zhang X D, Wang Y F, Zhou Z K, Zhang Y X, Wang X Z. Detection method for tomato leaf mildew based on hyperspectral fusion terahertz technology. Foods, 2023; 12(3): 535.

Wang Y F, Li T Z, Chen T H, Zhang X D, Taha M F, Yang N, et al. Cucumber downy mildew disease prediction using a CNN-LSTM approach. Agriculture, 2024; 14: 1155.

Zheng W G, Lan R P, Zhangzhong L, Yang L N, Gao L T, Yu J X. A hybrid approach for soil total nitrogen anomaly detection integrating machine learning and spatial statistics. Agronomy, 2023; 13(11): 2669.

Mangaiyarkarasi R. Aeroponics system for production of horticultural crops. Madras Agricultural Journal, 2020; 107: 1–7.

Kotzen B, Emerenciano M G, Moheimani N, Burnell G M. Aquaponics: Alternative types and approaches. In: Goddek S, Joyce A, Kotzen B, Burnell G M, editors. Aquaponics food production systems: Combined aquaculture and hydroponic production technologies for the future, Springer. 2019; pp.301–330.

Riggio G M, Jones S L, Gibson K E. Risk of human pathogen internalization in leafy vegetables during lab-scale hydroponic cultivation. Horticulturae, 2019; 5(1): 25.




Copyright (c) 2025 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office