Research Papers: Design Automation

Expensive Black-Box Model Optimization Via a Gold Rush Policy

[+] Author and Article Information
Benson Isaac

Department of Mechanical Engineering,
Texas A&M University,
College Station, TX 77843
e-mail: isaacbn@tamu.edu

Douglas Allaire

Department of Mechanical Engineering,
Texas A&M University,
College Station, TX 77843
e-mail: dallaire@tamu.edu

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received June 18, 2018; final manuscript received November 16, 2018; published online January 10, 2019. Assoc. Editor: Gary Wang.

J. Mech. Des 141(3), 031401 (Jan 10, 2019) (9 pages) Paper No: MD-18-1458; doi: 10.1115/1.4042113 History: Received June 18, 2018; Revised November 16, 2018

The optimization of black-box models is a challenging task owing to the lack of analytic gradient information and structural information about the underlying function, and also due often to significant run times. A common approach to tackling such problems is the implementation of Bayesian global optimization techniques. However, these techniques often rely on surrogate modeling strategies that endow the approximation of the underlying expensive function with nonexistent features. Further, these techniques tend to push new queries away from previously queried design points, making it difficult to locate an optimum point that rests near a previous model evaluation. To overcome these issues, we propose a gold rush (GR) policy that relies on purely local information to identify the next best design alternative to query. The method employs a surrogate constructed pointwise, that adds no additional features to the approximation. The result is a policy that performs well in comparison to state of the art Bayesian global optimization methods on several benchmark problems. The policy is also demonstrated on a constrained optimization problem using a penalty method.

Copyright © 2019 by ASME
Your Session has timed out. Please sign back in to continue.


Durantin, C. , Marzat, J. , and Balesdent, M. , 2016, “ Analysis of Multi-Objective Kriging-Based Methods for Constrained Global Optimization,” Comput. Optim. Appl., 63(3), pp. 903–926.
Sasena, M. J. , Papalambros, P. , and Goovaerts, P. , 2002, “ Exploration of Metamodeling Sampling Criteria for Constrained Global Optimization,” Eng. Optim., 34(3), pp. 263–278.
Ghoreishi, S. F. , and Allaire, D. L. , 2018, “ A Fusion-Based Multi-Information Source Optimization Approach Using Knowledge Gradient Policies,” AIAA Paper No. 2018-1159.
Parr, J. , Holden, C. M. , Forrester, A. I. , and Keane, A. J. , 2010, “ Review of Efficient Surrogate Infill Sampling Criteria With Constraint Handling,” Second International Conference on Engineering Optimization, Lisbon, Portugal, Sept. 6–9, pp. 1–10. https://pdfs.semanticscholar.org/df63/8813760971a61daaa91259ff65b817d0dd51.pdf
Shan, S. , and Wang, G. G. , 2010, “ Metamodeling for High Dimensional Simulation-Based Design Problems,” ASME J. Mech. Des., 132(5), p. 051009.
Forrester, A. I. , and Keane, A. J. , 2009, “ Recent Advances in Surrogate-Based Optimization,” Prog. Aerosp. Sci., 45(1–3), pp. 50–79.
Booker, A. J. , Dennis, J. E. , Frank, P. D. , Serafini, D. B. , Torczon, V. , and Trosset, M. W. , 1999, “ A Rigorous Framework for Optimization of Expensive Functions by Surrogates,” Struct. Optim., 17(1), pp. 1–13.
MoŠkus J.,1975, “On Bayesian Methods for Seeking the Extremum,” Optimization Techniques IFIP Technical Conference, Novosibirsk, July 1–7, pp. 400–405.
Žilinskas, A. , 1992, “ A Review of Statistical Models for Global Optimization,” J. Global Optim., 2(2), pp. 145–153.
Rasmussen, C. E. , 2004, “ Gaussian Processes in Machine Learning,” Advanced Lectures on Machine Learning. ML 2003. Lecture Notes in Computer Science, In: Bousquet O., von Luxburg U., Rätsch G., eds., Vol. 3176, Springer, Berlin, pp. 63–71.
Osborne, M. A. , Garnett, R. , and Roberts, S. J. , 2009, “ Gaussian Processes for Global Optimization,” Third International Conference on Learning and Intelligent Optimization (LION3), Trento, Italy, Jan. 14–18, pp. 1–15.
Pandita, P. , Bilionis, I. , and Panchal, J. , 2016, “ Extending Expected Improvement for High-Dimensional Stochastic Optimization of Expensive Black-Box Functions,” ASME J. Mech. Des., 138(11), p. 111412.
Koullias, S. , and Mavris, D. N. , 2014, “ Methodology for Global Optimization of Computationally Expensive Design Problems,” ASME J. Mech. Des., 136(8), p. 081007.
Jones, D. R. , Schonlau, M. , and Welch, W. J. , 1998, “ Efficient Global Optimization of Expensive Black-Box Functions,” J. Global Optim., 13(4), pp. 455–492.
Huang, D. , Allen, T. T. , Notz, W. I. , and Miller, R. A. , 2006, “ Sequential Kriging Optimization Using Multiple-Fidelity Evaluations,” Struct. Multidiscip. Optim., 32(5), pp. 369–382.
Humphrey, D. G. , and Wilson, J. R. , 2000, “ A Revised Simplex Search Procedure for Stochastic Simulation Response Surface Optimization,” INFORMS J. Comput., 12(4), pp. 272–283.
Borror, C. M. , Montgomery, D. C. , and Myers, R. H. , 2002, “ Evaluation of Statistical Designs for Experiments Involving Noise Variables,” J. Qual. Technol., 34(1), p. 54.
Gablonsky, J. M. , and Kelley, C. T. , 2001, “ A Locally-Biased Form of the Direct Algorithm,” J. Global Optim., 21(1), pp. 27–37.
Villemonteix, J. , Vazquez, E. , and Walter, E. , 2009, “ An Informational Approach to the Global Optimization of Expensive-to-Evaluate Functions,” J. Global Optim., 44(4), p. 509.
Kushner, H. J. , and Schweppe, F. C. , 1964, “ A Maximum Principle for Stochastic Control Systems,” J. Math. Anal. Appl., 8(2), pp. 287–302.
Jones, D. R. , 2001, “ A Taxonomy of Global Optimization Methods Based on Response Surfaces,” J. Global Optim., 21(4), pp. 345–383.
Shimoyama, K. , Sato, K. , Jeong, S. , and Obayashi, S. , 2013, “ Updating Kriging Surrogate Models Based on the Hypervolume Indicator in Multi-Objective Optimization,” ASME J. Mech. Des., 135(9), p. 094503.
Brochu, E. , Cora, V. M. , and De Freitas, N. , 2010, “ A Tutorial on Bayesian Optimization of Expensive Cost Functions, With Application to Active User Modeling and Hierarchical Reinforcement Learning,” e-print arXiv:1012.2599. https://arxiv.org/abs/1012.2599
Lai, T. L. , and Robbins, H. , 1985, “ Asymptotically Efficient Adaptive Allocation Rules,” Adv. Appl. Math., 6(1), pp. 4–22.
Hernández-Lobato, J. M. , Hoffman, M. W. , and Ghahramani, Z. , 2014, “ Predictive Entropy Search for Efficient Global Optimization of Black-Box Functions,” Adv. Neutral Inf. Process. Syst., 1, pp. 918–926. https://pdfs.semanticscholar.org/fbce/d739237b1bd07a6f3fb627b4c948751ca659.pdf
Moore, R. A. , Romero, D. A. , and Paredis, C. J. , 2014, “ Value-Based Global Optimization,” ASME J. Mech. Des., 136(4), p. 041003.
Thompson, S. C. , and Paredis, C. J. , 2010, “ An Investigation Into the Decision Analysis of Design Process Decisions,” ASME J. Mech. Des., 132(12), p. 121009.
Frazier, P. I. , Powell, W. B. , and Dayanik, S. , 2008, “ A Knowledge-Gradient Policy for Sequential Information Collection,” SIAM J. Control Optim., 47(5), pp. 2410–2439.
Gupta, S. S. , and Miescke, K. J. , 1996, “ Bayesian Look Ahead One-Stage Sampling Allocations for Selection of the Best Population,” J. Stat. Plann. Inference, 54(2), pp. 229–244.
Frazier, P. , Powell, W. , and Dayanik, S. , 2009, “ The Knowledge-Gradient Policy for Correlated Normal Beliefs,” INFORMS J. Comput., 21(4), pp. 599–613.
Negoescu, D. M. , Frazier, P. I. , and Powell, W. B. , 2011, “ The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery,” INFORMS J. Comput., 23(3), pp. 346–363.
Wu, J. , Poloczek, M. , Wilson, A. G. , and Frazier, P. , 2017, “ Bayesian Optimization With Gradients,” eprint arXiv:1703.04389 https://arxiv.org/abs/1703.04389
Andrianakis, I. , and Challenor, P. G. , 2012, “ The Effect of the Nugget on Gaussian Process Emulators of Computer Models,” Comput. Stat. Data Anal., 56(12), pp. 4215–4228.
Vapnik, V. , 1998, Statistical Learning Theory, Wiley, New York.
Jakeman, J. D. , and Wildey, T. , 2015, “ Enhancing Adaptive Sparse Grid Approximations and Improving Refinement Strategies Using Adjoint-Based a Posteriori Error Estimates,” J. Comput. Phys., 280, pp. 54–71.
Rabitz, H. , and Aliş, Ö. F. , 1999, “ General Foundations of High-Dimensional Model Representations,” J. Math. Chem., 25(2/3), pp. 197–233.
Li, K. , and Allaire, D. , 2016, “ A Compressed Sensing Approach to Uncertainty Propagation for Approximately Additive Functions,” ASME Paper No. DETC2016-60195.
Gorodetsky, A. A. , Karaman, S. , and Marzouk, Y. M. , 2015, “ Function-Train: A Continuous Analogue of the Tensor-Train Decomposition,” eprint arXiv:1510.09088. https://arxiv.org/abs/1510.09088
Allaire, D. , and Willcox, K. , 2010, “ Surrogate Modeling for Uncertainty Assessment With Application to Aviation Environmental System Models,” AIAA J., 48(8), pp. 1791–1803.
Amaral, S. , Allaire, D. , and Willcox, K. , 2017, “ Optimal L2-norm Empirical Importance Weights for the Change of Probability Measure,” Stat. Comput., 27(3), pp. 625–643.
Altman, N. S. , 1992, “ An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression,” Am. Statistician, 46(3), pp. 175–185.
Molga, M. , and Smutnicki, C. , 2005, “ Test Functions for Optimization Needs,” Comput. Inform. Sci., 01, pp. 1–43. http://new.zsd.iiar.pwr.wroc.pl/files/docs/functions.pdf
Branin, F. H. , 1972, “ Widely Convergent Method for Finding Multiple Solutions of Simultaneous Nonlinear Equations,” IBM J. Res. Dev., 16(5), pp. 504–522.
Huang, D. , Allen, T. T. , Notz, W. I. , and Zeng, N. , 2006, “ Global Optimization of Stochastic Black-Box Systems Via Sequential Kriging Meta-Models,” J. Global Optim., 34(3), pp. 441–466.
Ray, T. , 2003, “ Golinski's Speed Reducer Problem Revisited,” AIAA J., 41(3), pp. 556–558.
Hassan, R. , Cohanim, B. , De Weck, O. , and Venter, G. , 2005, “ A Comparison of Particle Swarm Optimization and the Genetic Algorithm,” AIAA Paper No. 2005-1897.
Li, H. , and Papalambros, P. , 1985, “ A Production System for Use of Global Optimization Knowledge,” J. Mech., Transm., Autom. Des., 107(2), pp. 277–284.
Ku, K. J. , Rao, S. , and Chen, L. , 1998, “ Taguchi-Aided Search Method for Design Optimization of Engineering Systems,” Eng. Optim., 30(1), pp. 1–23.
Kanukolanu, D. , Lewis, K. E. , and Winer, E. H. , 2006, “ A Multidimensional Visualization Interface to Aid in Trade-Off Decisions During the Solution of Coupled Subsystems Under Uncertainty,” ASME J. Comput. Inf. Sci. Eng., 6(3), pp. 288–299.
McKay, M. D. , Beckman, R. J. , and Conover, W. J. , 1979, “ Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code,” Technometrics, 21(2), pp. 239–245.
De Boer, P.-T. , Kroese, D. P. , Mannor, S. , and Rubinstein, R. Y. , 2005, “ A Tutorial on the Cross-Entropy Method,” Ann. Oper. Res., 134(1), pp. 19–67.
Glover, F. , 1989, “ Tabu Search—Part I,” ORSA J. Comput., 1(3), pp. 190–206.


Grahic Jump Location
Fig. 1

A graphical depiction of the computation of the weights required for the pointwise construction of a surrogate model of the expensive black-box function to be optimized

Grahic Jump Location
Fig. 2

Depiction of the LI for a one-dimensional function with three evaluations of the expensive black-box function

Grahic Jump Location
Fig. 3

A depiction of a few queries from an expensive black-box one-dimensional function being used to make pointwise surrogate approximations (top plot) and estimate the GR index (bottom plot)

Grahic Jump Location
Fig. 4

Graphs of the true Michalewicz function (left), a Gaussian process regression surrogate (middle), and a pointwise surrogate (right). The surrogate models were constructed using the same 5 × 5 grid of data.

Grahic Jump Location
Fig. 5

Convergence results for the CKG and GR policies for the Michalewicz, six-hump camel back, and Tilted Branin functions with an initial 3 × 3 grid

Grahic Jump Location
Fig. 6

Querying paths for the GR and CKG policies on the Tilted Branin function

Grahic Jump Location
Fig. 7

Golinski's speed reducer [46]

Grahic Jump Location
Fig. 8

Convergence results for the GR policy for a three-dimensional version of Golinski's speed reducer



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In