0
Research Papers: Design Automation

Error Metrics and the Sequential Refinement of Kriging Metamodels

[+] Author and Article Information
David A. Romero

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: d.romero@utoronto.ca

Veronica E. Marin

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: v.marin@mail.utoronto.ca

Cristina H. Amon

Mechanical & Industrial Engineering,
University of Toronto,
Toronto, ON M5S 3G8, Canada
e-mail: cristina.amon@utoronto.ca

Also referred to in the literature as data points, sampling points, or just points. Herein, we will refer to them as samples or sampling points.

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received April 12, 2013; final manuscript received October 10, 2014; published online November 14, 2014. Assoc. Editor: Shinji Nishiwaki.

J. Mech. Des 137(1), 011402 (Jan 01, 2015) (13 pages) Paper No: MD-13-1166; doi: 10.1115/1.4028883 History: Received April 12, 2013; Revised October 10, 2014; Online November 14, 2014

Metamodels, or surrogate models, have been proposed in the literature to reduce the resources (time/cost) invested in the design and optimization of engineering systems whose behavior is modeled using complex computer codes, in an area commonly known as simulation-based design optimization. Following the seminal paper of Sacks et al. (1989, “Design and Analysis of Computer Experiments,” Stat. Sci., 4(4), pp. 409–435), researchers have developed the field of design and analysis of computer experiments (DACE), focusing on different aspects of the problem such as experimental design, approximation methods, model fitting, model validation, and metamodeling-based optimization methods. Among these, model validation remains a key issue, as the reliability and trustworthiness of the results depend greatly on the quality of approximation of the metamodel. Typically, model validation involves calculating prediction errors of the metamodel using a data set different from the one used to build the model. Due to the high cost associated with computer experiments with simulation codes, validation approaches that do not require additional data points (samples) are preferable. However, it is documented that methods based on resampling, e.g., cross validation (CV), can exhibit oscillatory behavior during sequential/adaptive sampling and model refinement, thus making it difficult to quantify the approximation capabilities of the metamodels and/or to define rational stopping criteria for the metamodel refinement process. In this work, we present the results of a simulation experiment conducted to study the evolution of several error metrics during sequential model refinement, to estimate prediction errors, and to define proper stopping criteria without requiring additional samples beyond those used to build the metamodels. Our results show that it is possible to accurately estimate the predictive performance of Kriging metamodels without additional samples, and that leave-one-out CV errors perform poorly in this context. Based on our findings, we propose guidelines for choosing the sample size of computer experiments that use sequential/adaptive model refinement paradigm. We also propose a stopping criterion for sequential model refinement that does not require additional samples.

Copyright © 2015 by ASME
Topics: Errors
Your Session has timed out. Please sign back in to continue.

References

Figures

Grahic Jump Location
Fig. 1

Evolution of true metamodel errors versus dimensionality for two different sampling schemes, test functions based on Gaussian kernels

Grahic Jump Location
Fig. 2

Evolution of true metamodel errors versus sampling scheme for two different input dimensions, test functions based on Gaussian kernel

Grahic Jump Location
Fig. 3

Evolution of true metamodel errors versus sampling scheme for two different input dimensions, test functions based on exponential kernel

Grahic Jump Location
Fig. 4

Comparison of the evolution of TRUE and CV1 error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 5

Comparison of the evolution of TRUE and CV1 error metrics for two different input dimensions, test functions based on the exponential kernel

Grahic Jump Location
Fig. 6

Comparison of the evolution of TRUE and PSE error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 7

Comparison of the evolution of TRUE and MSE error metrics for two different input dimensions, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 8

Comparison of the evolution of TRUE and MSE error metrics for two different input dimensions, test functions based on the exponential kernel

Grahic Jump Location
Fig. 9

Evolution of error metrics for a particular case (Gaussian-based test function of 57, d = 3, ni /d = 4)

Grahic Jump Location
Fig. 10

Success rate of the (PSE ≤ 10%) stopping criteria as a function of total number of samples in the model, test functions based on the Gaussian kernel

Grahic Jump Location
Fig. 11

Success rate of the (PSE ≤ 10%) stopping criteria as a function of total number of samples in the model, test functions based on the exponential kernel

Grahic Jump Location
Fig. 12

Test function number 7/100, Gaussian kernel

Grahic Jump Location
Fig. 13

Test function number 16/100, Gaussian kernel

Grahic Jump Location
Fig. 14

Test function number 31/100, Gaussian kernel

Grahic Jump Location
Fig. 15

Test function number 38/100, Gaussian kernel

Grahic Jump Location
Fig. 16

Test function number 1/100, exponential kernel

Grahic Jump Location
Fig. 17

Test function number 21/100, exponential kernel

Grahic Jump Location
Fig. 18

Test function number 26/100, exponential kernel

Grahic Jump Location
Fig. 19

Test function number 32/100, exponential kernel

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In