"Representations of Quasi-Newton Matrices and their use in Limited Memory Methods"
R. Byrd, J. Nocedal, R. Schnabel.
Mathematical Programming, Vol. 63, No. 4, pp. 129-156 (1994)
In this article I will attempt to review the most recent advances in the theory of unconstrained optimization, and will also describe some important open questions. Before doing so, I should point out that the value of the theory of optimization is not limited to its capacity for explaining the behavior of the most widely used techniques. What do we know about the behavior of the most popular algorithms? How useful is the theory when designing new algorithms? i.e. how well can it differentiate between efficient and inefficient methods. Some interesting analysis will be discussed in this regard. We will see that the weaknesses of several classical algorithms that have fallen out of grace, such as the Fletcher-Reeves conjugate gradient method and the Davidon-Fletcher-Powell variable metric method, are fairly well understood. I will also describe several theoretical studies on optimization methods that have not yet enjoyed widespread popularity, but that may prove to be highly successful in the future.