ROBUST QUASI-NEWTON EQUATIONS IN QUASI-NEWTON METHOD FOR SOLVING UNCONSTRAINED OPTIMIZATION PROBLEMS
Abstract
Quasi-Newton methods are among the most widely used and effective general-purpose algorithms for unconstrained optimization. These methods traditionally rely on the quasi-Newton equation, which serves as the foundation for updating approximations of the Hessian matrix at each iteration. The goal is to construct accurate second-order curvature information to accelerate convergence toward the optimum. In this paper, we derive a novel quasi-Newton equation based on an enhanced quadratic model. A key feature of this new formulation is that it incorporates both gradient information and objective function values, enabling higher-order accuracy in approximating the second-order curvature of the objective function. This new equation stands out for its ability to provide a more precise representation of the function's curvature, which in turn improves the overall efficiency and performance of the optimization method. Theoretical analysis shows that the proposed method is globally convergent under certain reasonable assumptions. To validate the effectiveness of the approach, we conducted a series of numerical experiments using standard benchmark problems. The results demonstrate that the modified Broyden, Fletcher, Goldfarb, and Shanno (BFGS) method, which integrates the new quasi-Newton equation, outperforms existing BFGS-type methods in terms of numerical efficiency and solution accuracy.
Downloads
References
S. S. Boyd and L. Vandenberghe, Convex optimization Boyd, S. S., & Vandenberghe, L. (2004). CONVEX OPTIMIZATION. OPTIMIZATION METHODS AND SOFTWARE (Vol. 25). Cambridge University Press., vol. 25, no. 3. 2004.doi: http://doi.org/10.1080/10556781003625177
Jorge Nocedal and Stephen Wright, NUMERICAL OPTIMIZATION (Jorge Nocedal, Stephen Wright). 2006.
Fletcher R., PRACTICAL METHOD OF OPTIMIZATION, 2nd Edition. New York, 1989.
R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, “A LIMITED MEMORY ALGORITHM FOR BOUND CONSTRAINED OPTIMIZATION,” SIAM Journal on Scientific Computing, vol. 16, no. 5, 1995, doi: https://doi.org/10.1137/0916069.
D. C. Liu and J. Nocedal, “ON THE LIMITED MEMORY BFGS METHOD FOR LARGE SCALE OPTIMIZATION,” Math Program, vol. 45, no. 1–3, 1989, doi: https://doi.org/10.1007/BF01589116.
L. Bottou, F. E. Curtis, and J. Nocedal, “OPTIMIZATION METHODS FOR LARGE-SCALE MACHINE LEARNING,” 2018. doi: https://doi.org/10.1137/16M1080173.
E. Kahya, “MODIFIED SECANT-TYPE METHODS FOR UNCONSTRAINED OPTIMIZATION,” Appl Math Comput, vol. 181, no. 2, 2006, doi: https://doi.org/10.1016/j.amc.2006.03.003.
WOLFE P, “CONVERGENCE CONDITIONS FOR ASCENT METHODS,” SIAM Review, vol. 11, no. 2, pp. 226–235, Apr. 1969, doi: https://doi.org/10.1137/1011036.
S. S. Rao, ENGINEERING OPTIMIZATION: THEORY AND PRACTICE. 2019. doi: https://doi.org/10.1002/9781119454816.
R. H. Byrd and J. Nocedal, “A TOOL FOR THE ANALYSIS OF QUASI-NEWTON METHODS WITH APPLICATION TO UNCONSTRAINED MINIMIZATION,” SIAM J Numer Anal, vol. 26, no. 3, 1989, doi: https://doi.org/10.1137/0726042.
B. A. Hassan and A. A. Saad, “ELASTIC CONJUGATE GRADIENT METHODS TO SOLVE ITERATION PROBLEMS,” J. Interdiscip. Math., vol. 26, no. 6, pp. 1207–1217, 2023, doi: https://doi.org/10.47974/JIM-1619
M. J. Powell, “SOME GLOBAL CONVERGENCE PROPERTIES OF A VARIABLE METRIC ALGORITHM FOR MINIMIZATION WITHOUT EXACT LINE SEARCHES,” SIAM-AMS Proceedings, vol. 9, no. 1, 1976.
Y. H. Dai, “CONVERGENCE PROPERTIES OF THE BFGS ALGORITM,” SIAM Journal on Optimization, vol. 13, no. 3, 2003, doi: https://doi.org/10.1137/S1052623401383455
B. A. Hassan, “A NEW TYPE OF QUASI-NEWTON UPDATING FORMULAS BASED ON THE NEW QUASI-NEWTON EQUATION,” Numerical Algebra, Control and Optimization, vol. 10, no. 2, 2020, doi: https://doi.org/10.3934/naco.2019049.
B. A. Hassan and M. A. Kahya, “A NEW CLASS OF QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION,” Journal of Interdisciplinary Mathematics, vol. 24, no. 8, 2021, doi: https://doi.org/10.1080/09720502.2021.1961980.
B. A. Hassan and M. W. Taha, “A NEW VARIANTS OF QUASI-NEWTON EQUATION BASED ON THE QUADRATIC FUNCTION FOR UNCONSTRAINED OPTIMIZATION,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 19, no. 2, 2020, doi: https://doi.org/10.11591/ijeecs.v19.i2.pp701-708.
B. A. Hassan and R. M. Sulaiman, “USING A NEW TYPE QUASI-NEWTON EQUATION FOR UNCONSTRAINED OPTIMIZATION,” in Proceedings of the 7th International Engineering Conference “Research and Innovation Amid Global Pandemic”, IEC 2021, 2021. doi: https://doi.org/10.1109/IEC52205.2021.9476089.
B. A. Hassan and A. R. Ayoob, “AN ADAPTIVE QUASI-NEWTON EQUATION FOR UNCONSTRAINED OPTIMIZATION,” in Proceedings of 2021 2nd Information Technology to Enhance E-Learning and other Application Conference, IT-ELA 2021, 2021. doi: https://doi.org/10.1109/IT-ELA52201.2021.9773580.
J. Z. Zhang, N. Y. Deng, and L. H. Chen, “NEW QUASI-NEWTON EQUATION AND RELATED METHODS FOR UNCONSTRAINED OPTIMIZATION,” J Optim Theory Appl, vol. 102, no. 1, 1999, doi: https://doi.org/10.1023/A:1021898630001.
B. A. Hassan, F. Alfarag, and S. Djordjevic, “NEW STEP SIZES OF THE GRADIENT METHODS FOR UNCONSTRAINED OPTIMIZATION PROBLEm,” 2021.
B. A. Hassan and R. M. Sulaiman, “USING A NEW TYPE OF FORMULA CONJUGATE ON THE GRADIENT METHODS,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 27, no. 1, 2022, doi: https://doi.org/10.11591/ijeecs.v27.i1.pp86-91.
Sulaiman Ranen M., Abdullah Zeyad M., and Hassan Basim A., “NEW FORMULA ON THE CONJUGATE GRADIENT METHOD FOR UNCONSTRAINED OPTIMIZATION AND ITS APPLICATION,” Iraqi Journal of Science, vol. 65, no. 9, pp. 5182–5194, 2024, doi: https://doi.org/10.24996/ijs.2024.65.9.32.
G. Yuan, Z. Sheng, B. Wang, W. Hu, and C. Li, “THE GLOBAL CONVERGENCE OF A MODIFIED BFGS METHOD FOR NONCONVEX FUNCTIONS,” J Comput Appl Math, vol. 327, 2018, doi: 10.1016/j.cam.2017.05.030.
G. Yuan, Z. Wei, and Y. Wu, “MODIFIED LIMITED MEMORY BFGS METHOD WITH NONMONOTONE LINE SEARCH FOR UNCONSTRAINED OPTIMIZATION,” Journal of the Korean Mathematical Society, vol. 47, no. 4, 2010, doi: https://doi.org/10.4134/JKMS.2010.47.4.767.
E. D. Dolan and J. J. Moré, “BENCHMARKING OPTIMIZATION SOFTWARE WITH PERFORMANCE PROFILES,” Mathematical Programming, Series B, vol. 91, no. 2, 2002, doi: https://doi.org/10.1007/s101070100263.
D. H. Li and M. Fukushima, “A MODIFIED BFGS METHOD AND ITS GLOBAL CONVERGENCE IN NONCONVEX MINIMIZATION,” J Comput Appl Math, vol. 129, no. 1–2, 2001, doi: https://doi.org/10.1016/S0377-0427(00)00540-9.
Z. Wei, G. Li, and L. Qi, “NEW QUASI-NEWTON METHODS FOR UNCONSTRAINED OPTIMIZATION PROBLEMS,” Appl Math Comput, vol. 175, no. 2, 2006, doi: https://doi.org/10.1016/j.amc.2005.08.027.
B. A. Hassan and G. M. Al-Naemi, “A NEW QUASI-NEWTON EQUATION ON THE GRADIENT METHODS FOR OPTIMIZATION MINIMIZATION PROBLEM,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 19, no. 2, 2020, doi: https://doi.org/10.11591/ijeecs.v19.i2.pp737-744.
B. A. Hassan and I. A. R. Moghrabi, “A MODIFIED SECANT EQUATION QUASI-NEWTON METHOD FOR UNCONSTRAINED OPTIMIZATION,” J Appl Math Comput, vol. 69, no. 1, 2023, doi: https://doi.org/10.1007/s12190-022-01750-x.
Basim A. Hassan and Abdulrahman R. Ayoob, “ON THE NEW QUASI-NEWTON EQUATION FOR UNCONSTRAINED OPTIMIZATION,” 8th IEC 2022 - International Engineering Conference: Towards Engineering Innovations and Sustainability, 2022.doi: https://doi.org/10.1109/IEC54822.2022.9807584
Basim A. Hassan and Hameed M. Sadiq, (2022), A NEW FORMULA ON THE CONJUGATE GRADIENT METHOD FOR REMOVING IMPULSE NOISE IMAGES, Bulletin of the South Ural State University. Ser. Mathematical Modelling, Programming & Computer Software (Bulletin SUSU MMCS), 2022, vol. 15, no. 4, pp. 123-130. doi: https://doi.org/10.14529/mmp220412
Hassan, B. A., & Sadiq, H. (2022). EFFICIENT NEW CONJUGATE GRADIENT METHODS FOR REMOVING IMPULSE NOISE IMAGES. European Journal of Pure and Applied Mathematics, 15(4), 2011–2021.doi: https://doi.org/10.29020/nybg.ejpam.v15i4.4568
Copyright (c) 2026 Basim A. Hassan, Manal I. Mohammed
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this Journal agree to the following terms:
- Author retain copyright and grant the journal right of first publication with the work simultaneously licensed under a creative commons attribution license that allow others to share the work within an acknowledgement of the work’s authorship and initial publication of this journal.
- Authors are able to enter into separate, additional contractual arrangement for the non-exclusive distribution of the journal’s published version of the work (e.g. acknowledgement of its initial publication in this journal).
- Authors are permitted and encouraged to post their work online (e.g. in institutional repositories or on their websites) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published works.
1.gif)
