0
Research Papers

Toward a Better Understanding of Model Validation Metrics

[+] Author and Article Information
Yu Liu

Visiting Predoctoral Student at Northwestern University School of Mechatronics Engineering,  University of Electronic Science and Technology of China, Chengdu 611731, China

Wei Chen1

Department of Mechanical Engineering,  Northwestern University, 2145 Sheridan Road, Tech B224, Evanston, IL 60208weichen@northwestern.edu

Paul Arendt

Department of Mechanical Engineering,  Northwestern University, Evanston, IL 60208

Hong-Zhong Huang

School of Mechatronics Engineering,  University of Electronic Science and Technology of China, Chengdu 611731, China

1

Corresponding author.

J. Mech. Des 133(7), 071005 (Jul 08, 2011) (13 pages) doi:10.1115/1.4004223 History: Received October 22, 2010; Revised April 03, 2011; Published July 07, 2011; Online July 08, 2011

Model validation metrics have been developed to provide a quantitative measure that characterizes the agreement between predictions and observations. In engineering design, the metrics become useful for model selection when alternative models are being considered. Additionally, the predictive capability of a computational model needs to be assessed before it is used in engineering analysis and design. Due to the various sources of uncertainties in both computer simulations and physical experiments, model validation must be conducted based on stochastic characteristics. Currently there is no unified validation metric that is widely accepted. In this paper, we present a classification of validation metrics based on their key characteristics along with a discussion of the desired features. Focusing on stochastic validation with the consideration of uncertainty in both predictions and physical experiments, four main types of metrics, namely classical hypothesis testing, Bayes factor, frequentist’s metric, and area metric, are examined to provide a better understanding of the pros and cons of each. Using mathematical examples, a set of numerical studies are designed to answer various research questions and study how sensitive these metrics are with respect to the experimental data size, the uncertainty from measurement error, and the uncertainty in unknown model parameters. The insight gained from this work provides useful guidelines for choosing the appropriate validation metric in engineering applications.

FIGURES IN THIS ARTICLE
<>
Copyright © 2011 by American Society of Mechanical Engineers
Your Session has timed out. Please sign back in to continue.

References

Figures

Grahic Jump Location
Figure 1

Illustration of the u-pooling method. (a) u-values at multiple validation sites; (b) area metric of mismatch between the empirical distribution of u-values and the standard uniform distribution.

Grahic Jump Location
Figure 2

Physical observations (Exp) versus model predictions (Pred) for test set 1

Grahic Jump Location
Figure 3

Physical observations (Exp) versus model predictions (Pred) for test set 2

Grahic Jump Location
Figure 4

Percentage of rejecting model versus the number of observations at validation site x = 6.0 in test set 1

Grahic Jump Location
Figure 5

Percentage of rejecting model versus the number of observations at validation site x = 6.0 in test set 2

Grahic Jump Location
Figure 6

Distributions of log(B0 ) at validation site x = 6.0

Grahic Jump Location
Figure 7

Impact from the amount of physical observations at x = 6.0

Grahic Jump Location
Figure 8

95% confidence bounds of estimated error versus the number of physical observations at x = 6.0 in test set 1

Grahic Jump Location
Figure 9

Distributions of area metrics at validation site x = 6.0 in test set 1

Grahic Jump Location
Figure 10

Distributions of global metrics (u-pooling metrics) in test set 1

Grahic Jump Location
Figure 11

Distributions of area metrics at validation site x = 6.0 for test set 2

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In