Starting with the fundamentals of classical smooth optimization and building on established convex programming techniques, this research monograph presents a foundation and methodology for modern nonconvex nondifferentiable optimization. It provides readers with theory, methods, and applications of nonconvex and nondifferentiable optimization in statistical estimation, operations research, machine learning, and decision making. A comprehensive and rigorous treatment of this emergent mathematical topic is urgently needed in today’s complex world of big data and machine learning. This book takes a thorough approach to the subject and includes examples and exercises to enrich the main themes, making it suitable for classroom instruction. Modern Nonconvex Nondifferentiable Optimization is intended for applied and computational mathematicians, optimizers, operations researchers, statisticians, computer scientists, engineers, economists, and machine learners. It could be used in advanced courses on optimization/operations research and nonconvex and nonsmooth optimization.

"This monograph serves present and future needs where nonconvexity and nondifferentiability are inevitably present in the faithful modeling of real-world applications of optimization"--

A popular way to assess the “effort” needed to solve a problem is to count how many evaluations of the problem functions (and their derivatives) are required. In many cases, this is often the dominating computational cost. Given an optimization problem satisfying reasonable assumptions—and given access to problem-function values and derivatives of various degrees—how many evaluations might be required to approximately solve the problem? Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation, and Perspectives addresses this question for nonconvex optimization problems, those that may have local minimizers and appear most often in practice. This is the first book on complexity to cover topics such as composite and constrained optimization, derivative-free optimization, subproblem solution, and optimal (lower and sharpness) bounds for nonconvex problems. It is also the first to address the disadvantages of traditional optimality measures and propose useful surrogates leading to algorithms that compute approximate high-order critical points, and to compare traditional and new methods, highlighting the advantages of the latter from a complexity point of view. This is the go-to book for those interested in solving nonconvex optimization problems. It is suitable for advanced undergraduate and graduate students in courses on advanced numerical analysis, data science, numerical optimization, and approximation theory.

Optimization is one of the most important areas of modern applied mathematics, with applications in fields from engineering and economics to finance, statistics, management science, and medicine. While many books have addressed its various aspects, Nonlinear Optimization is the first comprehensive treatment that will allow graduate students and researchers to understand its modern ideas, principles, and methods within a reasonable time, but without sacrificing mathematical precision. Andrzej Ruszczynski, a leading expert in the optimization of nonlinear stochastic systems, integrates the theory and the methods of nonlinear optimization in a unified, clear, and mathematically rigorous fashion, with detailed and easy-to-follow proofs illustrated by numerous examples and figures. The book covers convex analysis, the theory of optimality conditions, duality theory, and numerical methods for solving unconstrained and constrained optimization problems. It addresses not only classical material but also modern topics such as optimality conditions and numerical methods for problems involving nondifferentiable functions, semidefinite programming, metric regularity and stability theory of set-constrained systems, and sensitivity analysis of optimization problems. Based on a decade's worth of notes the author compiled in successfully teaching the subject, this book will help readers to understand the mathematical foundations of the modern theory and methods of nonlinear optimization and to analyze new problems, develop optimality theory for them, and choose or construct numerical solution methods. It is a must for anyone seriously interested in optimization.

The contributions gathered in this open access book focus on modern methods for data science and classification and present a series of real-world applications. Numerous research topics are covered, ranging from statistical inference and modeling to clustering and dimension reduction, from functional data analysis to time series analysis, and network analysis. The applications reflect new analyses in a variety of fields, including medicine, marketing, genetics, engineering, and education. The book comprises selected and peer-reviewed papers presented at the 17th Conference of the International Federation of Classification Societies (IFCS 2022), held in Porto, Portugal, July 19–23, 2022. The IFCS federates the classification societies and the IFCS biennial conference brings together researchers and stakeholders in the areas of Data Science, Classification, and Machine Learning. It provides a forum for presenting high-quality theoretical and applied works, and promoting and fostering interdisciplinary research and international cooperation. The intended audience is researchers and practitioners who seek the latest developments and applications in the field of data science and classification.

Non-convex Optimization for Machine Learning takes an in-depth look at the basics of non-convex optimization with applications to machine learning. It introduces the rich literature in this area, as well as equips the reader with the tools and techniques needed to apply and analyze simple but powerful procedures for non-convex problems. Non-convex Optimization for Machine Learning is as self-contained as possible while not losing focus of the main topic of non-convex optimization techniques. The monograph initiates the discussion with entire chapters devoted to presenting a tutorial-like treatment of basic concepts in convex analysis and optimization, as well as their non-convex counterparts. The monograph concludes with a look at four interesting applications in the areas of machine learning and signal processing, and exploring how the non-convex optimization techniques introduced earlier can be used to solve these problems. The monograph also contains, for each of the topics discussed, exercises and figures designed to engage the reader, as well as extensive bibliographic notes pointing towards classical works and recent advances. Non-convex Optimization for Machine Learning can be used for a semester-length course on the basics of non-convex optimization with applications to machine learning. On the other hand, it is also possible to cherry pick individual portions, such the chapter on sparse recovery, or the EM algorithm, for inclusion in a broader course. Several courses such as those in machine learning, optimization, and signal processing may benefit from the inclusion of such topics.

Built on the framework of the successful first edition, this book serves as a modern introduction to the field of optimization. The author’s objective is to provide the foundations of theory and algorithms of nonlinear optimization as well as to present a variety of applications from diverse areas of applied sciences. Introduction to Nonlinear Optimization gradually yet rigorously builds connections between theory, algorithms, applications, and actual implementation. The book contains several topics not typically included in optimization books, such as optimality conditions in sparsity constrained optimization, hidden convexity, and total least squares. Readers will discover a wide array of applications such as circle fitting, Chebyshev center, the Fermat–Weber problem, denoising, clustering, total least squares, and orthogonal regression. These applications are studied both theoretically and algorithmically, illustrating concepts such as duality. Python and MATLAB programs are used to show how the theory can be implemented. The extremely popular CVX toolbox (MATLAB) and CVXPY module (Python) are described and used. More than 250 theoretical, algorithmic, and numerical exercises enhance the reader's understanding of the topics. (More than 70 of the exercises provide detailed solutions, and many others are provided with final answers.) The theoretical and algorithmic topics are illustrated by Python and MATLAB examples. This book is intended for graduate or advanced undergraduate students in mathematics, computer science, electrical engineering, and potentially other engineering disciplines.

Moment and polynomial optimization is an active research field used to solve difficult questions in many areas, including global optimization, tensor computation, saddle points, Nash equilibrium, and bilevel programs, and it has many applications. The author synthesizes current research and applications, providing a systematic introduction to theory and methods, a comprehensive approach for extracting optimizers and solving truncated moment problems, and a creative methodology for using optimality conditions to construct tight Moment-SOS relaxations. This book is intended for applied mathematicians, engineers, and researchers entering the field. It can be used as a textbook for graduate students in courses on convex optimization, polynomial optimization, and matrix and tensor optimization.

This concise, self-contained volume introduces convex analysis and optimization algorithms, with an emphasis on bridging the two areas. It explores cutting-edge algorithms—such as the proximal gradient, Douglas–Rachford, Peaceman–Rachford, and FISTA—that have applications in machine learning, signal processing, image reconstruction, and other fields. An Introduction to Convexity, Optimization, and Algorithms contains algorithms illustrated by Julia examples and more than 200 exercises that enhance the reader’s understanding of the topic. Clear explanations and step-by-step algorithmic descriptions facilitate self-study for individuals looking to enhance their expertise in convex analysis and optimization. Designed for courses in convex analysis, numerical optimization, and related subjects, this volume is intended for undergraduate and graduate students in mathematics, computer science, and engineering. Its concise length makes it ideal for a one-semester course. Researchers and professionals in applied areas, such as data science and machine learning, will find insights relevant to their work.

The only book offering solved exercises for integer and combinatorial optimization, this book contains 102 classroom tested problems of varying scope and difficulty chosen from a plethora of topics and applications. It has an associated website containing additional problems, lecture notes, and suggested readings. Topics covered include modeling capabilities of integer variables, the Branch-and-Bound method, cutting planes, network optimization models, shortest path problems, optimum tree problems, maximal cardinality matching problems, matching-covering duality, symmetric and asymmetric TSP, 2-matching and 1-tree relaxations, VRP formulations, and dynamic programming. Problems and Solutions for Integer and Combinatorial Optimization: Building Skills in Discrete Optimization is meant for undergraduate and beginning graduate students in mathematics, computer science, and engineering to use for self-study and for instructors to use in conjunction with other course material and when teaching courses in discrete optimization.