0
Research Papers: Design Automation

A New Interval Area Metric for Model Validation With Limited Experimental Data

[+] Author and Article Information
Ning Wang

College of Aerospace Science and Engineering,
National University of Defense Technology,
Changsha 410073, China
e-mail: wangning14@nudt.edu.cn

Wen Yao

College of Aerospace Science and Engineering,
National University of Defense Technology,
Changsha 410073, China
e-mail: yaowen@nudt.edu.cn

Yong Zhao

College of Aerospace Science and Engineering,
National University of Defense Technology,
Changsha 410073, China
e-mail: zhaoyong@nudt.edu.cn

Xiaoqian Chen

College of Aerospace Science and Engineering,
National University of Defense Technology,
Changsha 410073, China
e-mail: chenxiaoqian@nudt.edu.cn

Xiang Zhang

College of Aerospace Science and Engineering,
National University of Defense Technology,
Changsha 410073, China
e-mail: zxstudy@hotmail.com

Lanmin Li

China Academy of Space Technology,
Institute of Space Electronic Technology,
Shandong 264670, China

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received October 18, 2017; final manuscript received March 14, 2018; published online April 17, 2018. Assoc. Editor: Mian Li.

J. Mech. Des 140(6), 061403 (Apr 17, 2018) (11 pages) Paper No: MD-17-1699; doi: 10.1115/1.4039689 History: Received October 18, 2017; Revised March 14, 2018

Various stochastic validation metrics have been developed for validating models, among which area metric is frequently used in many practical problems. However, the existing area metric does not consider experimental epistemic uncertainty caused by lack of sufficient physical observations. Therefore, it cannot provide a confidence level associated with the amount of experimental data, which is a desired characteristic of validation metric. In this paper, the concept of area metric is extended to a new metric, namely interval area metric, for single-site model validation with limited experimental data. The kernel of the proposed metric is defining two boundary distribution functions based on Dvoretzky–Kiefer–Wolfowitz inequality, so as to provide an interval at a given confidence level, which covers the true cumulative distribution function (CDF) of physical observations. Based on this interval area metric, the validity of a model can be quantitatively measured with the specific confidence level in association with consideration of the lack of experiment information. The new metric is examined and compared with the existing metrics through numerical case studies to demonstrate its validity and discover its properties. Furthermore, an engineering example is provided to illustrate the effectiveness of the proposed metric in practical satellite structure engineering application.

FIGURES IN THIS ARTICLE
<>
Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.

References

Malak, R. J. , and Paredis, C. J. J. , 2010, “ Using Support Vector Machines to Formalize the Valid Input Domain of Predictive Models in Systems Design Problems,” ASME J. Mech. Des., 132(10), p. 101001. [CrossRef]
Oberkampf, W. L. , Sindir, M. N. , and Terrence Conlisk, A. , 1998, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” American Institute of Aeronautics and Astronautics, Reston, VA, AIAA Standard No. G-077-1998.
Liu, Y. , Chen, W. , Arendt, P. , and Huang, H. Z. , 2011, “ Toward a Better Understanding of Model Validation Metrics,” ASME J. Mech. Des., 133(7), p. 071005. [CrossRef]
Ferson, S. , Oberkampf, W. L. , and Ginzburg, L. , 2008, “ Model Validation and Predictive Capability for the Thermal Challenge Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2408–2430. [CrossRef]
Choudhary, A. , Voyles, I. T. , Roy, C. J. , Oberkampf, W. L. , and Patil, M. , 2016, “ Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem,” ASME J. Verif. Validation Uncertainty Quantif., 1(1), p. 011003. [CrossRef]
Li, W. , Chen, S. , Jiang, Z. , Apley, D. W. , Lu, Z. , and Chen, W. , 2016, “ Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem,” ASME J. Verif. Validation Uncertainty Quantif., 1(1), p. 011004. [CrossRef]
Xi, Z. , and Yang, R.-J. , 2015, “ Reliability Analysis With Model Uncertainty Coupling With Parameter and Experiment Uncertainties: A Case Study of 2014 Verification and Validation Challenge Problem,” ASME J. Verif. Validation Uncertainty Quantif., 1(1), p. 011005. [CrossRef]
Lima, E. A. B. F. , Oden, J. T. , Wohlmuth, B. , Shahmoradi, A. , Hormuth, D. A. , Yankeelov, T. E. , Scarabosio, L. , and Horger, T. , 2017, “ Selection and Validation of Predictive Models of Radiation Effects on Tumor Growth Based on Noninvasive Imaging Data,” Comput. Methods Appl. Mech. Eng., 327, pp. 277–305. [CrossRef] [PubMed]
Jung, B. C. , Park, J. , Oh, H. , Kim, J. , and Youn, B. D. , 2015, “ A Framework of Model Validation and Virtual Product Qualification With Limited Experimental Data Based on Statistical Inference,” Struct. Multidiscip. Optim., 51(3), pp. 573–583. [CrossRef]
Wang, Z. , Fu, Y. , Yang, R. J. , Barbat, S. , and Chen, W. , 2016, “ Validating Dynamic Engineering Models Under Uncertainty,” ASME J. Mech. Des., 138(11), p. 111402. [CrossRef]
Ling, Y. , and Mahadevan, S. , 2013, “ Quantitative Model Validation Techniques: New Insights,” Reliab. Eng. Syst. Saf., 111, pp. 217–231. [CrossRef]
Voyles, I. T. , and Roy, C. J. , 2014, “ Evaluation of Model Validation Techniques in the Presence of Uncertainty,” AIAA Paper No. 2014-0120.
Li, W. , Chen, W. , Jiang, Z. , Lu, Z. , and Liu, Y. , 2014, “ New Validation Metrics for Models With Multiple Correlated Responses,” Reliab. Eng. Syst. Saf., 127, pp. 1–11. [CrossRef]
Pradlwarter, H. J. , and Schuëller, G. I. , 2008, “ The Use of Kernel Densities and Confidence Intervals to Cope With Insufficient Data in Validation Experiments,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2550–2560. [CrossRef]
Ao, D. , Hu, Z. , and Mahadevan, S. , 2017, “ Dynamics Model Validation Using Time-Domain Metrics,” ASME J. Verif. Validation Uncertainty Quantif., 2(1), p. 011004. [CrossRef]
Roy, C. J. , and Oberkampf, W. L. , 2011, “ A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing,” Comput. Methods Appl. Mech. Eng., 200(25–28), pp. 2131–2144. [CrossRef]
Beer, M. , Ferson, S. , and Kreinovich, V. Y. , 2013, “ Imprecise Probabilities in Engineering Analyses,” Mech. Syst. Signal Process., 37(1–2), pp. 4–29. [CrossRef]
Oberkampf, W. L. , and Roy, C. J. , 2010, Verification and Validation in Scientific Computing, Cambridge University Press, Cambridge, UK. [CrossRef]
Ferson, S. , Kreinovick, V. , Ginzburg, L. , and Sentz, F. , 2003, “ Constructing Probability Boxes and Dempster-Shafer Structures,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2002-4015.
Van der Vaart, A. W. , 1998, Asymptotic Statistics, Cambridge University Press, Cambridge, UK. [CrossRef] [PubMed] [PubMed]
Xiong, Y. , Chen, W. , Tsui, K. L. , and Apley, D. W. , 2009, “ A Better Understanding of Model Updating Strategies in Validating Engineering Models,” Comput. Methods Appl. Mech. Eng., 198(15–16), pp. 1327–1337. [CrossRef]
Oliver, T. A. , Terejanu, G. , Simmons, C. S. , and Moser, R. D. , 2015, “ Validating Predictions of Unobserved Quantities,” Comput. Methods Appl. Mech. Eng., 283, pp. 1310–1335. [CrossRef]

Figures

Grahic Jump Location
Fig. 1

Three cases of relationship between CDF of prediction and boundary distribution functions of observation: (a) case 1, (b) case 2, and (c) case 3

Grahic Jump Location
Fig. 2

Flowchart of validation procedure using interval area metric

Grahic Jump Location
Fig. 3

Comparison of average area difference interval, distribution of original area metric values, and true area values in test 1 (σε=0.08)

Grahic Jump Location
Fig. 4

Comparison of average area difference interval, distribution of original area metric values, and true area values in test 1 (σε=0.2)

Grahic Jump Location
Fig. 5

Comparison of average area difference interval, distribution of original area metric values, and true area values in test 2

Grahic Jump Location
Fig. 6

Box plots of observations and predictions at three validation sites

Grahic Jump Location
Fig. 7

Area difference interval versus number of observations for three cases

Grahic Jump Location
Fig. 8

Length of area difference interval versus number of observations for three cases

Grahic Jump Location
Fig. 9

Length of area difference interval versus confidence level for three cases

Grahic Jump Location
Fig. 10

Comparison of D-K-W limits and K-S limits of EDF of observations at the confidence level 90%

Grahic Jump Location
Fig. 11

Comparison of average area difference interval, distribution of original area metric values, and distribution of limiting values obtained by reliability metric

Grahic Jump Location
Fig. 12

The cantilever beam example for satellite support

Grahic Jump Location
Fig. 13

Setup of experiment

Grahic Jump Location
Fig. 14

Cumulative distribution functions of prediction from model 1 and model 2, and the boundary distribution functions of observation

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In