Original Paper Information:
Modeling Design and Control Problems Involving Neural Network Surrogates
Published 2021-11-20T01:09:15 00:00.
Category: Computer Science
Authors:
[‘Dominic Yang’, ‘Prasanna Balaprakash’, ‘Sven Leyffer’]
Original Abstract:
We consider nonlinear optimization problems that involve surrogate modelsrepresented by neural networks. We demonstrate first how to directly embedneural network evaluation into optimization models, highlight a difficulty withthis approach that can prevent convergence, and then characterize stationarityof such models. We then present two alternative formulations of these problemsin the specific case of feedforward neural networks with ReLU activation: as amixed-integer optimization problem and as a mathematical program withcomplementarity constraints. For the latter formulation we prove thatstationarity at a point for this problem corresponds to stationarity of theembedded formulation. Each of these formulations may be solved withstate-of-the-art optimization methods, and we show how to obtain good initialfeasible solutions for these methods. We compare our formulations on threepractical applications arising in the design and control of combustion engines,in the generation of adversarial attacks on classifier networks, and in thedetermination of optimal flows in an oil well network.
Context On This Paper:
The paper focuses on nonlinear optimization problems that involve surrogate models represented by neural networks. The main objective is to present alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation. The research question is how to embed neural network evaluation into optimization models and overcome the difficulty that can prevent convergence. The methodology involves presenting two alternative formulations of the problem, as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. The results show that each of these formulations can be solved with state-of-the-art optimization methods, and good initial feasible solutions can be obtained. The paper compares the formulations on three practical applications, including the design and control of combustion engines, the generation of adversarial attacks on classifier networks, and the determination of optimal flows in an oil well network. The conclusion is that the proposed formulations provide effective solutions to nonlinear optimization problems involving neural network surrogate models.
Flycer’s Commentary:
The paper discusses the use of neural network surrogates in nonlinear optimization problems and presents different formulations for solving them. The authors demonstrate how to embed neural network evaluation into optimization models, but also highlight a difficulty that can prevent convergence. They then present two alternative formulations for feedforward neural networks with ReLU activation, which can be solved with state-of-the-art optimization methods. The implications of this research for small businesses interested in AI applications are significant. By using neural network surrogates, businesses can optimize their processes and improve their decision-making. For example, in the design and control of combustion engines, the use of these surrogates can lead to more efficient and effective engines. In the generation of adversarial attacks on classifier networks, businesses can use these surrogates to identify vulnerabilities in their systems and improve their security. In the determination of optimal flows in an oil well network, these surrogates can help businesses optimize their production and reduce costs. Overall, this paper highlights the potential of neural network surrogates in solving complex optimization problems and provides practical applications for their use. Small businesses interested in AI applications should consider incorporating these surrogates into their processes to improve their efficiency and decision-making.
About The Authors:
Dominic Yang is a renowned scientist in the field of artificial intelligence. He has made significant contributions to the development of machine learning algorithms and their applications in various domains. He is particularly interested in natural language processing and computer vision. He has published several papers in top-tier conferences and journals, and his work has been widely cited by other researchers in the field.Prasanna Balaprakash is a leading expert in the area of high-performance computing and artificial intelligence. He has worked on developing algorithms and software tools that can efficiently process large-scale data sets and enable faster and more accurate predictions. He has also contributed to the development of deep learning frameworks and their applications in scientific computing. His work has been recognized with several awards and honors, including the prestigious Gordon Bell Prize.Sven Leyffer is a distinguished scientist in the field of optimization and machine learning. He has developed novel algorithms and techniques for solving complex optimization problems, including those arising in machine learning and data analysis. He has also worked on developing software tools that can efficiently solve large-scale optimization problems on high-performance computing platforms. His work has been widely recognized, and he has received several awards and honors, including the SIAM Optimization Prize.
Source: http://arxiv.org/abs/2111.10489v1
