A THREE-TERM CONJUGATE GRADIENT METHOD FOR LARGE-SCALE MINIMIZATION IN ARTIFICIAL NEURAL NETWORKS
Abstract
Conjugate Gradient (CG) methods are widely used for solving unconstrained optimization problems due to their efficiency and low memory requirements. However, standard CG methods may not always guarantee sufficient descent condition, which can impact their robustness and convergence behavior. Additionally, their effectiveness in training artificial neural networks (ANNs) remains an area of interest. In response, this paper presents a three-term conjugate gradient (CG) method for unconstrained optimization problems. The new parameter is formulated so that the search direction satisfies the sufficient descent condition. The global convergence result of the new algorithm is discussed under suitable assumptions. To evaluate the performance of the new method we considered some standard test problems for unconstrained optimization and applied the proposed method to train different ANNs on some benchmark data sets contained in the NN toolbox. The experimental results show that performance is encouraging for both unconstrained minimization test problems and in training neural networks.
Downloads
References
V. Dhaka and M. P. Singh, “HANDWRITTEN CHARACTER RECOGNITION USING MODIFIED GRADIENT DESCENT TECHNIQUE OF NEURAL NETWORKS AND REPRESENTATION OF CONJUGATE DESCENT FOR TRAINING PATTERNS,” Int. J. Eng., vol. 22, no. 2, pp. 145–158, Jun. 2009.
N. Sato, K. Izumi, and H. Iiduka, “SCALED CONJUGATE GRADIENT METHOD FOR NONCONVEX OPTIMIZATION IN DEEP NEURAL NETWORKS,” J. Mach. Learn. Res., vol. 25, no. 395, pp. 1–37, 2024.
Z. Yang, “ADAPTIVE STOCHASTIC CONJUGATE GRADIENT FOR MACHINE LEARNING,” Expert Syst. Appl., vol. 206, p. 117719, Nov. 2022, doi: https://doi.org/10.1016/j.eswa.2022.117719.
W. Huang and L. Ju, “A NOVEL INTRUSION DETECTION METHOD BASED ON CONJUGATE GRADIENT NEURAL NETWORK,” in 2010 International Conference On Computer Design and Applications, Jun. 2010, pp. V2-470-V2-472. doi: https://doi.org/10.1109/ICCDA.2010.5541362.
H. Zhu, Y. Xiao, and S.-Y. Wu, “LARGE SPARSE SIGNAL RECOVERY BY CONJUGATE GRADIENT ALGORITHM BASED ON SMOOTHING TECHNIQUE,” Comput. Math. Appl., vol. 66, no. 1, pp. 24–32, Aug. 2013, doi: https://doi.org/10.1016/j.camwa.2013.04.022.
I. M. Sulaiman, M. Malik, W. Giyarti, M. Mamat, M. A. H. Ibrahim, and M. Z. Ahmad, “The Application of Conjugate Gradient Method to Motion Control of Robotic Manipulators,” in Enabling Industry 4.0 through Advances in Mechatronics, Eds., Singapore: Springer Nature, 2022, pp. 435–445. doi: https://doi.org/10.1007/978-981-19-2095-0_37.
B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A SURVEY OF MOTION PLANNING AND CONTROL TECHNIQUES FOR SELF-DRIVING URBAN VEHICLES,” IEEE Trans. Intell. Veh., vol. 1, no. 1, pp. 33–55, Mar. 2016, doi: https://doi.org/10.1109/TIV.2016.2578706.
I. M. Navon and D. M. Legler, “CONJUGATE-GRADIENT METHODS FOR LARGE-SCALE MINIMIZATION IN METEOROLOGY,” Aug. 1987, doi: https://doi.org/10.1175/1520-0493(1987)115<1479:CGMFLS>2.0.CO;2
A. Muhammed Awwal, P. Kumam, K. Sitthithakerngkiet, A. Muhammad, A. Halilu, and S. Ibrahim, “DERIVATIVE-FREE METHOD BASED ON DFP UPDATING FORMULA FOR SOLVING CONVEX CONSTRAINED NONLINEAR MONOTONE EQUATIONS AND APPLICATION,” AIMS Math., vol. 6, pp. 8792–8814, Jun. 2021, doi: 10.3934/math.2021510.
S. Ibrahim, M. Mamat, A. Abashar, M. Rivaie, and Z. Salleh, “A MODIFIED NONLINEAR CONJUGATE GRADIENT METHOD FOR UNCONSTRAINED OPTIMIZATION,” Appl. Math. Sci., vol. 9, pp. 2671–2682, Jan. 2015, doi: https://doi.org/10.12988/ams.2015.5141.
M. Malik, M. Mamat, S. Abas, S. Ibrahim, and S. Firman, “PERFORMANCE ANALYSIS OF NEW SPECTRAL AND HYBRID CONJUGATE GRADIENT METHODS FOR SOLVING UNCONSTRAINED OPTIMIZATION PROBLEMS,” IAENG Int. J. Comput. Sci., vol. 48, pp. 66–79, Feb. 2021.
R. Fletcher and C. M. Reeves, “FUNCTION MINIMIZATION BY CONJUGATE GRADIENTS,” Comput. J., vol. 7, no. 2, pp. 149–154, Jan. 1964, doi: https://doi.org/10.1093/comjnl/7.2.149.
E. Polak and G. Ribiere, “NOTE SUR LA CONVERGENCE DE MÉTHODES DE DIRECTIONS CONJUGUÉES,” Rev. Fr. Inform. Rech. Opérationnelle Sér. Rouge, vol. 3, no. 16, Art. no. 16, 1969, doi: https://doi.org/10.1051/m2an/196903R100351.
B. T. Polyak, “THE CONJUGATE GRADIENT METHOD IN EXTREMAL PROBLEMS,” USSR Comput. Math. Math. Phys., vol. 9, no. 4, pp. 94–112, Jan. 1969, doi: https://doi.org/10.1016/0041-5553(69)90035-4.
M. R. Hestenes and E. Stiefel, “METHODS OF CONJUGATE GRADIENTS FOR SOLVING LINEAR SYSTEMS,” J. Res. Natl. Bur. Stand., vol. 49, no. 6, p. 409, Dec. 1952, doi: https://doi.org/10.6028/jres.049.044.
Y. H. Dai and Y. Yuan, “A NONLINEAR CONJUGATE GRADIENT METHOD WITH A STRONG GLOBAL CONVERGENCE PROPERTY,” SIAM J. Optim., vol. 10, no. 1, pp. 177–182, Jan. 1999, doi: https://doi.org/10.1137/S1052623497318992.
Y. Liu and C. Storey, “EFFICIENT GENERALIZED CONJUGATE GRADIENT ALGORITHMS, PART 1: THEORY,” J. Optim. Theory Appl., vol. 69, no. 1, pp. 129–137, Apr. 1991, doi: https://doi.org/10.1007/BF00940464.
M. Malik, S. S. Abas, M. Mamat, Sukono, and I. S. Mohammed, “A NEW HYBRID CONJUGATE GRADIENT METHOD WITH GLOBAL CONVERGENCE PROPERTIES,” Int. J. Adv. Sci. Technol., vol. 29, no. 05, Art. no. 05, Apr. 2020.
S. M. Ibrahim, N. Idalisa, N. Zullpakkal, P. Kaelo, and N. Shapiee, “A DESCENT THREE-TERM CONJUGATE GRADIENT DIRECTION FOR PROBLEMS OF UNCONSTRAINED OPTIMIZATION WITH APPLICATION,” Int. J. Comput. Methods, Feb. 2025, doi: https://doi.org/10.1142/S0219876225500100.
A. Mousavi, M. Esmaeilpour, and A. Sheikhahmadi, “TWO EFFICIENT THREE-TERM CONJUGATE GRADIENT METHODS FOR IMPULSE NOISE REMOVAL FROM MEDICAL IMAGES,” Multimed. Tools Appl., vol. 83, no. 15, pp. 43685–43703, May 2024, doi: https://doi.org/10.1007/s11042-023-17352-z.
A. Alhawarat, Z. Salleh, H. Alolaiyan, H. El Hor, and S. Ismail, “A THREE-TERM CONJUGATE GRADIENT DESCENT METHOD WITH SOME APPLICATIONS,” J. Inequalities Appl., vol. 2024, no. 1, p. 73, May 2024, doi: https://doi.org/10.1186/s13660-024-03142-0.
J. BARZILAI and J. M. BORWEIN, “TWO-POINT STEP SIZE GRADIENT METHODS,” IMA J. Numer. Anal., vol. 8, no. 1, pp. 141–148, Jan. 1988, doi: https://doi.org/10.1093/imanum/8.1.141.
I. M. Sulaiman, Sukono, S. Supian, and M. Mamat, “NEW CLASS OF HYBRID CONJUGATE GRADIENT COEFFICIENTS WITH GUARANTEED DESCENT AND EFFICIENT LINE SEARCH,” IOP Conf. Ser. Mater. Sci. Eng., vol. 621, no. 1, p. 012021, Oct. 2019, doi: https://doi.org/10.1088/1757-899X/621/1/012021.
N. Andrei, “A SIMPLE THREE-TERM CONJUGATE GRADIENT ALGORITHM FOR UNCONSTRAINED OPTIMIZATION,” J. Comput. Appl. Math., vol. 241, pp. 19–29, Mar. 2013, doi: https://doi.org/10.1016/j.cam.2012.10.002.
Y. -H. Dai and L. -Z. Liao, “NEW CONJUGACY CONDITIONS AND RELATED NONLINEAR CONJUGATE GRADIENT METHODS,” Appl. Math. Optim., vol. 43, no. 1, pp. 87–101, Jan. 2001, doi: https://doi.org/10.1007/s002450010019.
R. Fletcher, “A NEW APPROACH TO VARIABLE METRIC ALGORITHMS,” Comput. J., vol. 13, no. 3, pp. 317–322, Jan. 1970, doi: https://doi.org/10.1093/comjnl/13.3.317.
B. E. M, “A DERIVATION OF CONJUGATE-GRADIENTS,” Numer. Methods Non-Linear Optim., 1972, Accessed: Feb. 26, 2025. [Online]. Available: https://cir.nii.ac.jp/crid/1573950399913297920
L. Nazareth, “A Conjugate Direction Algorithm Without Line Searches,” J. Optim. Theory Appl., vol. 23, no. 3, pp. 373–387, Nov. 1977, doi: https://doi.org/10.1007/BF00933447.
W. W. Hager and H. Zhang, “A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS”.
L. Zhang, W. Zhou, and D. Li, “SOME DESCENT THREE-TERM CONJUGATE GRADIENT METHODS AND THEIR GLOBAL CONVERGENCE,” Optim. Methods Softw., vol. 22, no. 4, pp. 697–711, Aug. 2007, doi: https://doi.org/10.1080/10556780701223293.
N. Deng and Z. Li, “GLOBAL CONVERGENCE OF THREE TERMS CONJUGATE GRADIENT METHODS ∗,” Optim. Methods Softw., Jan. 1994, doi: https://doi.org/10.1080/10556789508805593.
W. W. Hager and H. Zhang, “A NEW CONJUGATE GRADIENT METHOD WITH GUARANTEED DESCENT AND AN EFFICIENT LINE SEARCH,” SIAM J. Optim., vol. 16, no. 1, pp. 170–192, Jan. 2005, doi: https://doi.org/10.1137/030601880.
A. A. Goldstein, “ON STEEPEST DESCENT,” J. Soc. Ind. Appl. Math. Ser. Control, vol. 3, no. 1, pp. 147–151, Jan. 1965, doi: https://doi.org/10.1137/0303013.
A. L. Ibrahim and M. G. Mohammed, “A NEW THREE-TERM CONJUGATE GRADIENT METHOD FOR TRAINING NEURAL NETWORKS WITH GLOBAL CONVERGENCE,” Indones. J. Electr. Eng. Comput. Sci., vol. 28, no. 1, Art. no. 1, Oct. 2022, doi: https://doi.org/10.11591/ijeecs.v28.i1.pp551-558.
N. Andrei, “AN UNCONSTRAINED OPTIMIZATION TEST FUNCTIONS COLLECTION,” 2008. Accessed: Feb. 26, 2025. [Online]. Available: https://www.semanticscholar.org/paper/An-Unconstrained-Optimization-Test-Functions-Andrei/3bcfd42ec838188f3366d75d3568c0638a79c8c4
E. D. Dolan and J. J. Moré, “BENCHMARKING OPTIMIZATION SOFTWARE WITH PERFORMANCE PROFILES,” Math. Program., vol. 91, no. 2, pp. 201–213, Jan. 2002, doi: https://doi.org/10.1007/s101070100263.
K. Tu, S. Yang, H. Zhang, and Z. Wang, “HYBRID A∗ BASED MOTION PLANNING FOR AUTONOMOUS VEHICLES IN UNSTRUCTURED ENVIRONMENT,” in 2019 IEEE International Symposium on Circuits and Systems (ISCAS), May 2019, pp. 1–4. doi: https://doi.org/10.1109/ISCAS.2019.8702779.
Y. Zhang et al., “HYBRID TRAJECTORY PLANNING FOR AUTONOMOUS DRIVING IN HIGHLY CONSTRAINED ENVIRONMENTS,” IEEE Access, vol. 6, pp. 32800–32819, 2018, doi: 10 https://doi.org/10.1109/ACCESS.2018.2845448.
H. Zhang, Q. Qi, W. Ji, and F. Tao, “AN UPDATE METHOD FOR DIGITAL TWIN MULTI-DIMENSION MODELS,” Robot. Comput.-Integr. Manuf., vol. 80, p. 102481, Apr. 2023, doi: https://doi.org/10.1016/j.rcim.2022.102481.
T. Sutharssan, S. Stoyanov, C. Bailey, and C. Yin, “PROGNOSTIC AND HEALTH MANAGEMENT FOR ENGINEERING SYSTEMS: A REVIEW OF THE DATA-DRIVEN APPROACH AND ALGORITHMS,” J. Eng., vol. 2015, no. 7, pp. 215–222, 2015, doi: https://doi.org/10.1049/joe.2014.0303.
G. Ezzati, M. Mammadov, and S. Kulkarni, “A New Reliability Analysis Method Based On The Conjugate Gradient Direction,” Struct. Multidiscip. Optim., vol. 51, no. 1, pp. 89–98, Jan. 2015, doi: https://doi.org/10.1007/s00158-014-1113-z.
H. K. Seong, H. Shin, J. Yoo, T. Yamada, and S. Nishiwaki, “REACTION-DIFFUSION EQUATION BASED TOPOLOGY OPTIMIZATION COMBINED WITH THE MODIFIED CONJUGATE GRADIENT METHOD,” Finite Elem. Anal. Des., vol. 140, pp. 84–95, Feb. 2018, doi: https://doi.org/10.1016/j.finel.2017.11.009.
L. Montoli, “MULTI-OBJECTIVE TOPOLOGY OPTIMIZATION FOR CONJUGATE HEAT TRANSFER WITH THE ADJOINT METHOD,” Apr. 2021, Accessed: Feb. 26, 2025. [Online]. Available: https://www.politesi.polimi.it/handle/10589/174045
K. Uba kamfa, S. Ibrahim, M. Awang Nawi, M. Waziri, and M. Mamat, “PERFORMANCE EVALUATION OF A NOVEL CONJUGATE GRADIENT METHOD FOR TRAINING FEED FORWARD NEURAL NETWORK,” Math. Model. Comput., vol. 10, Apr. 2023, doi: 10.23939/mmc2023.02.326.
H. Kim et al., “VARIABLE THREE-TERM CONJUGATE GRADIENT METHOD FOR TRAINING ARTIFICIAL NEURAL NETWORKS,” Neural Netw., vol. 159, pp. 125–136, Feb. 2023, doi: https://doi.org/10.1016/j.neunet.2022.12.001.
I. Livieris and P. Pintelas, “Performance Evaluation of Descent CG Methods for Neural Network Training,” in HERMIS-μπ. Hellenic European Research on Mathematics and Informatics Science, 2009, pp. 40–46.
Copyright (c) 2025 Umar A Omesa, Muhammad Y. Waziri, Issam A. R. Moghrabi, Sulaiman M. Ibrahim, Gudu E B, Fakai S L, Rabiu Bashir Yunus, Elissa Nadia Madi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this Journal agree to the following terms:
- Author retain copyright and grant the journal right of first publication with the work simultaneously licensed under a creative commons attribution license that allow others to share the work within an acknowledgement of the work’s authorship and initial publication of this journal.
- Authors are able to enter into separate, additional contractual arrangement for the non-exclusive distribution of the journal’s published version of the work (e.g. acknowledgement of its initial publication in this journal).
- Authors are permitted and encouraged to post their work online (e.g. in institutional repositories or on their websites) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published works.
1.gif)
