This paper deals with generalized pattern search (GPS) algorithms for linearly constrained optimization. At each iteration, the GPS algorithm generates a set of directions that conforms to the geometry of any nearby linear constrains, and this is used to define the POLL set for that iteration. The contribution of this paper is to provide a detailed algorithm for constructing the set of directions at a current iterate whether or not the constraints are degenerate. The main difficulty in the degenerate case is in classifying constraints as redundant and nonredundant . We give a short survey of the main definitions and methods concerning redundancy and propose an approach, which may be useful for other active set algorithms, to identify the nonredundant constraints.

This paper deals with generalized pattern search (GPS) algorithms for linearly constrained optimization. At each iteration, the GPS algorithm generates a set of directions that conforms to the geometry of any nearby linear constraints. This set is then used to construct trial points to be evaluated during the iteration. In previous work, Lewis and Torczon developed a scheme for computing the conforming directions, but it assumed no degeneracy near the current iterate. The contribution of this paper is to provide a detailed algorithm for constructing the set of directions whether or not the constraints are degenerate. One difficulty in the degenerate case is in classifying constraints as redundant and nonredundant. We give a short survey of the main definitions and methods for treating redundancy and propose an approach to identify nonredundant "--Active constraints, which may be useful for other active set algorithms. We also introduce a new approach for handling nonredundant linearly dependent constraints, which maintains GPS convergence properties without significantly increasing computational cost. Some simple numerical tests illustrate the effectiveness of the algorithm. We conclude by briefly considering the extension of our ideas to nonlinear constraints with linearly dependent constraint gradients.

Abstract. Previous analyses of pattern search algorithms for unconstrained and linearly constrained minimization have focused on proving convergence of a subsequence of iterates to a limit point satisfying either directional or first-order necessary conditions for optimality, depending on the smoothness of the objective function in a neighborhood of the limit point. Even though pattern search methods require no derivative information, we are able to prove some limited directional second-order results. Although not as strong as classical second-order necessary conditions, these results are stronger than the first order conditions that many gradient-based methods satisfy. Under fairly mild conditions, we can eliminate from consideration all strict local maximizers and an entire class of saddle points.

This paper contains a new convergence analysis for the Lewis and Torezon GPS class of pattern search methods for linearly constrained optimization. The analysis is motivated by the desire to understand the behavior of the algorithm under hypotheses more consistent with properties satisfied in practice for a class of problems, discussed at various points in the paper, for which these methods are successful. Specifically, even if the objective function is discontinuous or extended valued, the methods find a limit point with some minimizing properties. Simple examples show that the strength of the optimality conditions at a limit point does not depend only on the algorithm, but also on the directions it uses, and on the smoothness of the objective at the limit point in question. This contribution of this paper is to provide a simple convergence analysis that supplies detail about the relation of optimality conditions to objective smoothness properties, and the defining directions for the algorithm, and it gives older results as easy corollaries.

A description of the implicit filtering algorithm, its convergence theory and a new MATLAB® implementation.

In the intervening years since this book was published in 1981, the field of optimization has been exceptionally lively. This fertility has involved not only progress in theory, but also faster numerical algorithms and extensions into unexpected or previously unknown areas such as semidefinite programming. Despite these changes, many of the important principles and much of the intuition can be found in this Classics version of Practical Optimization. This book provides model algorithms and pseudocode, useful tools for users who prefer to write their own code as well as for those who want to understand externally provided code. It presents algorithms in a step-by-step format, revealing the overall structure of the underlying procedures and thereby allowing a high-level perspective on the fundamental differences. And it contains a wealth of techniques and strategies that are well suited for optimization in the twenty-first century, and particularly in the now-flourishing fields of data science, “big data,” and machine learning. Practical Optimization is appropriate for advanced undergraduates, graduate students, and researchers interested in methods for solving optimization problems.