0
Research Papers: Design Automation

Error Metrics and the Sequential Refinement of Kriging Metamodels

[+] Author and Article Information
David A. Romero

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: d.romero@utoronto.ca

Veronica E. Marin

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: v.marin@mail.utoronto.ca

Cristina H. Amon

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: cristina.amon@utoronto.ca

Also referred to in the literature as data points, sampling points, or just points. Herein, we will refer to them as samples or sampling points.

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received April 12, 2013; final manuscript received October 10, 2014; published online November 14, 2014. Assoc. Editor: Shinji Nishiwaki.

J. Mech. Des 137(1), 011402 (Jan 01, 2015) (13 pages) Paper No: MD-13-1166; doi: 10.1115/1.4028883 History: Received April 12, 2013; Revised October 10, 2014; Online November 14, 2014

Metamodels, or surrogate models, have been proposed in the literature to reduce the resources (time/cost) invested in the design and optimization of engineering systems whose behavior is modeled using complex computer codes, in an area commonly known as simulation-based design optimization. Following the seminal paper of Sacks et al. (1989, “Design and Analysis of Computer Experiments,” Stat. Sci., 4(4), pp. 409–435), researchers have developed the field of design and analysis of computer experiments (DACE), focusing on different aspects of the problem such as experimental design, approximation methods, model fitting, model validation, and metamodeling-based optimization methods. Among these, model validation remains a key issue, as the reliability and trustworthiness of the results depend greatly on the quality of approximation of the metamodel. Typically, model validation involves calculating prediction errors of the metamodel using a data set different from the one used to build the model. Due to the high cost associated with computer experiments with simulation codes, validation approaches that do not require additional data points (samples) are preferable. However, it is documented that methods based on resampling, e.g., cross validation (CV), can exhibit oscillatory behavior during sequential/adaptive sampling and model refinement, thus making it difficult to quantify the approximation capabilities of the metamodels and/or to define rational stopping criteria for the metamodel refinement process. In this work, we present the results of a simulation experiment conducted to study the evolution of several error metrics during sequential model refinement, to estimate prediction errors, and to define proper stopping criteria without requiring additional samples beyond those used to build the metamodels. Our results show that it is possible to accurately estimate the predictive performance of Kriging metamodels without additional samples, and that leave-one-out CV errors perform poorly in this context. Based on our findings, we propose guidelines for choosing the sample size of computer experiments that use sequential/adaptive model refinement paradigm. We also propose a stopping criterion for sequential model refinement that does not require additional samples.

Copyright © 2015 by ASME
Topics: Errors
Your Session has timed out. Please sign back in to continue.

References

Queipo, N., Haftka, R., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P., 2005, “Surrogate-Based Analysis and Optimization,” Prog. Aerospace Sci., 41(1), pp. 1–28. [CrossRef]
Sacks, J., Welch, W., Mitchell, T., and Wynn, H., 1989, “Design and Analysis of Computer Experiments,” Stat. Sci., 4(4), pp. 409–435. [CrossRef]
Wang, G., and Shan, S., 2007, “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” ASME J. Mech. Des., 129(4), pp. 370–380. [CrossRef]
Forrester, A., and Keane, A., 2009, “Recent Advances in Surrogate-Based Optimization,” Prog. Aerospace Sci., 45(1–3), pp. 50–79 [CrossRef]
McKay, M., Beckman, R., and Conover, W., 1979, “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code,” Technometrics, 21(2), pp. 239–245.
Jones, D., Schonlau, M., and Welch, W., 1998, “Efficient Global Optimization of Expensive Black-Box Functions,” J. Global Optim., 13(4), pp. 455–492. [CrossRef]
Sasena, M., Papalambros, P., and Goovaerts, P., 2002, “Exploration of Metamodeling Sampling Criteria for Constrained Global Optimization,” Eng. Optim., 34(3), pp. 263–278. [CrossRef]
Jin, R., Chen, W., and Simpson, T., 2001, “Comparative Studies of Metamodeling Techniques Under Multiple Modeling Criteria,” Struct. Multidisc. Optim., 23(1), pp. 1–13. [CrossRef]
Goel, T., Haftka, R., and Shyy, W., 2009, “Comparing Error Estimation Measures for Polynomial and Kriging Approximation of Noise-Free Functions,” Struct. Multidisc. Optim., 38(5), pp. 429–442. [CrossRef]
Goel, T., and Stander, N., 2009, “Comparing Three Error Criteria for Selecting Radial Basis Function Network Topology,” Comput. Methods Appl. Mech. Eng., 198(27), pp. 2137–2150. [CrossRef]
Viana, F., Picheny, V., and Haftka, R., 2010, “Using Cross-Validation to Design Conservative Surrogates,” AIAA J., 48(10), pp. 2286–2298. [CrossRef]
Viana, F., Haftka, R., and Steffen, V., 2009, “Multiple Surrogates: How Cross-Validation Errors Can Help Us to Obtain the Best Predictor,” Struct. Multidisc. Optim., 39(4), pp. 439–457. [CrossRef]
Viana, F., and Haftka, R., 2009, “Cross Validation Can Estimate How Well Prediction Variance Correlates With Error,” AIAA J., 47(9), pp. 2266–2270. [CrossRef]
Loeppky, J., Sacks, J., and Welch, W., 2009, “Choosing the Sample Size of a Computer Experiment: A Practical Guide,” Technometrics, 51(4), pp. 366–376. [CrossRef]
Bischl, B., Mersmann, O., and Trautmann, H., 2010, “Resampling Methods in Model Validation,” Workshop on Experimental Methods for the Assessment of Computational Systems (WEMACS 2010), held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN 2010), Krakow, Poland, Sept. 11, p. 14.
Martin, J., 2009, “Computational Improvements to Estimating Kriging Metamodel Parameters,” ASME J. Mech. Des., 131(8), p. 084501. [CrossRef]
Lophaven, S., Nielsen, H., and Sondergaard, J., 2002, “Aspects of the matlab toolbox DACE,” Technical Report No. IMM-REP-2002-13.
Fang, K., Li, R., and Sudjianto, A., 2006, Design and Modeling for Computer Experiments, Chapman & Hall/CRC, Boca Raton, FL.
Jin, R., Chen, W., and Sudjianto, A., 2002, “On Sequential Sampling for Global Metamodeling in Engineering Design,” ASME Paper No. DETC2002/DAC-34092. [CrossRef]
Lin, Y., Mistree, F., Allen, J., Tsui, K., and Chen, V., 2004, “A Sequential Exploratory Experimental Design Method: Development of Appropriate Empirical Models in Design,” ASME Paper No. DETC2004-57527. [CrossRef]
Romero, D., Amon, C., and Finger, S., 2006, “On Adaptive Sampling for Single and Multi-Response Bayesian Surrogate Models,” ASME Paper No. DETC2006-99210. [CrossRef]
Wang, G., and Shan, S., 2006, “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” ASME Paper No. DETC2006-99412. [CrossRef]
Osio, I. G., 1996, “Multistage Bayesian Surrogates and Optimal Sampling for Engineering Design and Process Improvement,” Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.
Romero, D., Amon, C., and Finger, S., 2012, “Multiresponse Metamodeling in Simulation-Based Design Applications,” ASME J. Mech. Des., 134(9), p. 091001. [CrossRef]
Shewry, M., and Wynn, H., 1987, “Maximum Entropy Sampling,” J. Appl. Stat., 14(2), pp. 165–170. [CrossRef]
Sacks, J., Schiller, S., and Welch, W., 1989, “Designs for Computer Experiments,” Technometrics, 31(1), pp. 41–47. [CrossRef]
Santner, T., Williams, B., and Notz, W., 2003, The Design and Analysis of Computer Experiments (Springer Series in Statistics), Springer-Verlag, Inc., New York.
Martin, J., and Simpson, T., 2004, “On the Use of Kriging Models to Approximate Deterministic Computer Models,” ASME Paper No. DETC2004-57300. [CrossRef]
Emmerich, M., Giannakoglou, K., and Naujoks, B., 2006, “Single- and Multi-Objective Evolutionary Optimization Assisted by Gaussian Random Field Metamodels,” IEEE Trans. Evol. Comput., 10(4), pp. 421–439. [CrossRef]
Ginsbourger, D., Le Riche, R., and Carraro, L., 2007, “A Multi-Points Criterion for Deterministic Parallel Global Optimization Based on Kriging,” International Conference on Nonconvex Programming (NCP07) National Institute for Applied Sciences Rouen, France, Dec. 17–21.
Ponweiser, W., Wagner, T., and Vincze, M., 2008, “Clustered Multiple Generalized Expected Improvement: A Novel Infill Sampling Criterion for Surrogate Models,” IEEE Congress on Evolutionary Computation, pp. 1515–1522, Hong Kong, June 1–6, pp. 3515–3522. [CrossRef]
Wasserman, L., 2006, All of Nonparametric Statistics (Springer Texts in Statistics), Springer-Verlag, New York.
Meckesheimer, M., Booker, A., Barton, R., and Simpson, T., 2002, “Computationally Inexpensive Metamodel Assessment Strategies,” AIAA J., 40(10), pp. 2053–2060. [CrossRef]
Wasserman, L., 2006, All of Statistics: A Concise Course in Statistical Inference (Springer Texts in Statistics), Springer-Verlag, New York.
Trosset, M., and Padula, A., 2000, “Designing and Analyzing Computational Experiments for Global Optimization,” Technical Report No. TR00-25.
Burnham, K., and Anderson, D., 2004, “Multimodel Selection: Understanding AIC and BIC in Model Selection,” Sociolog. Meth. Res., 33(2), pp. 261–304.
Akaike, H., 1974. “A New Look at the Statistical Model Identification,” IEEE Trans. Autom. Control, AC-19(6), pp. 716–723. [CrossRef]

Figures

Grahic Jump Location
Fig. 1

Evolution of true metamodel errors versus dimensionality for two different sampling schemes, test functions based on Gaussian kernels

Grahic Jump Location
Fig. 2

Evolution of true metamodel errors versus sampling scheme for two different input dimensions, test functions based on Gaussian kernel

Grahic Jump Location
Fig. 3

Evolution of true metamodel errors versus sampling scheme for two different input dimensions, test functions based on exponential kernel

Grahic Jump Location
Fig. 4

Comparison of the evolution of TRUE and CV1 error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 5

Comparison of the evolution of TRUE and CV1 error metrics for two different input dimensions, test functions based on the exponential kernel

Grahic Jump Location
Fig. 6

Comparison of the evolution of TRUE and PSE error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 7

Comparison of the evolution of TRUE and MSE error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 8

Comparison of the evolution of TRUE and MSE error metrics for two different input dimensions, test functions based on the exponential kernel

Grahic Jump Location
Fig. 9

Evolution of error metrics for a particular case (Gaussian-based test function of 57, d = 3, ni /d = 4)

Grahic Jump Location
Fig. 10

Success rate of the (PSE ≤ 10%) stopping criteria as a function of total number of samples in the model, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 11

Success rate of the (PSE ≤ 10%) stopping criteria as a function of total number of samples in the model, test functions based on the exponential kernel

Grahic Jump Location
Fig. 12

Test function number 7/100, Gaussian kernel

Grahic Jump Location
Fig. 13

Test function number 16/100, Gaussian kernel

Grahic Jump Location
Fig. 14

Test function number 31/100, Gaussian kernel

Grahic Jump Location
Fig. 15

Test function number 38/100, Gaussian kernel

Grahic Jump Location
Fig. 16

Test function number 1/100, exponential kernel

Grahic Jump Location
Fig. 17

Test function number 21/100, exponential kernel

Grahic Jump Location
Fig. 18

Test function number 26/100, exponential kernel

Grahic Jump Location
Fig. 19

Test function number 32/100, exponential kernel

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In