Research Papers

Model Selection Among Physics-Based Models

[+] Author and Article Information
S. Mahadevan

e-mail: sankaran.mahadevan@vanderbilt.edu
Department of Civil and
Environmental Engineering,
Vanderbilt University
, Nashville, TN 37235

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received January 15, 2012; final manuscript received August 1, 2012; published online January 7, 2013. Assoc. Editor: Michael Kokkolaras.

J. Mech. Des 135(2), 021003 (Jan 07, 2013) (15 pages) Paper No: MD-12-1046; doi: 10.1115/1.4023155 History: Received January 15, 2012; Revised August 01, 2012

The optimal solution of a design optimization problem is dependent on the predictive models used to evaluate the objective and constraints. Since different models give different predictions and can yield different design decisions, when more than one model is available, the choice of model used to represent the objectives/constraints of the design becomes important. This paper addresses the problem of model selection among physics-based models during the prediction stage, which is in contrast to model selection during the calibration and validation stages, and therefore affects design under uncertainty. Model selection during calibration addresses the problem of selecting the model that is likely to provide the best generalization of the calibration data over the entire domain. Model selection during the validation stage examines the validity of a calibrated model by comparing its predictions against the validation data. This paper presents an approach that allows for model selection during the prediction stage, which selects the “best” model for each prediction point. The proposed approach is based on the estimation of the model prediction error under stationary/nonstationary uncertainty. By selecting the best model at each prediction point, the proposed approach partitions the input domain of the models into nonoverlapping regions. The effects of measurement noise, sparseness of validation data, and model prediction uncertainty are included in deriving a probabilistic selection criterion for model selection. The effects of these uncertainties on the classification errors are analyzed. The proposed approach is demonstrated for the problem of selecting between two parametric models for energy dissipation in a mechanical lap joint under dynamic loading, and for the problem of selecting fatigue crack growth models.

Copyright © 2013 by ASME
Your Session has timed out. Please sign back in to continue.


Kennedy, M. C., and O'Hagan, A., 2001, “Bayesian Calibration of Computer Models,” J. R. Stat. Soc. Ser. B (Stat. Methodol.), 63(3), pp. 425–464. [CrossRef]
Rebba, R., Mahadevan, S., and Huang, S., 2006, “Validation and Error Estimation of Computational Models,” Reliab. Eng. Syst. Saf., 91(10–11), pp. 1390–1397. [CrossRef]
American Society of Mechanical Engineers, V&V 20, 2009, Available at ASME Codes & Standards website.
Mahadevan, S., and Liang, B., 2011, “Error and Uncertainty Quantification and Sensitivity Analysis in Mechanics Computational Models,” Int. J. Uncertainty Quantification, 1(2), pp. 147–161. [CrossRef]
Roache, P. J., 1998, “Verification and Validation in Computational Science and Engineering,” Comput. Sci. Eng., Albuquerque, NM: Hermosa Publishers, 1998, pp. 8–9.
Rangavajhala, S., Sura, V. S., Hombal, V. K., and Mahadevan, S., 2011, “Discretization Error Estimation in Multidisciplinary Simulations,” AIAA J., 49(12), pp. 2673–2683. [CrossRef]
Akaike, H., 1978, “A Bayesian Analysis of the Minimum AIC Procedure,” Ann. Inst. Stat. Math., 30(1), pp. 9–14. [CrossRef]
Burnham, K. P., and Anderson, D. R., Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Springer-Verlag, Berlin.
Rissanen, J., 1978, “Modeling by Shortest Data Description,” Automatica, 14(5), pp. 465–471. [CrossRef]
Schwarz, G., 1978, “Estimating the Dimension of a Model,” Ann. Stat., 6(2), pp. 461–464. [CrossRef]
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and Van Der Linde, A., 2002, “Bayesian Measures of Model Complexity and Fit,” J. R. Stat. Soc. Ser. B (Stat. Methodol.), 64, pp. 583–639. [CrossRef]
Liu, Y., Chen, W., Arendt, P., and Huang, H.-Z., 2011, “Toward a Better Understanding of Model Validation Metrics,” ASME J. Mech. Des., 133(7), p. 071005. [CrossRef]
Oberkampf, W. L., and Roy, C. J., 2010, Verification and Validation in Scientific Computing, Cambridge University Press, Cambridge.
Rebba, R., and Mahadevan, S., 2008, “Computational Methods for Model Reliability Assessment,” Reliab. Eng. Syst. Saf., 93(8), pp. 1197–1207. [CrossRef]
Romero, V. J., 2011, “Elements of a Pragmatic Approach for Dealing With Bias and Uncertainty in Experiments Through Predictions,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2011-7342.
Hills, R. G., and Leslie, I., 2003, “Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2003-0706.
Oberkampf, W. L., and Barone, M. F., 2006, “Measures of Agreement Between Computation and Experiment: Validation Metrics,” J. Comput. Phys., 217(1), pp. 5–36. [CrossRef]
Urbina, A., and Paez, T. L., 2003, “Statistical Validation of Structural Dynamics Models,” J. IEST, 46(1), pp. 141–148. Available at http://iest.metapress.com/content/p3k33743858u56hx/?p=2987fd7493ee4efea050d89b4f290d98&pi=9
Zhang, R., and Mahadevan, S., 2003, “Bayesian Methodology for Reliability Model Acceptance,” Reliab. Eng. Syst. Saf., 80(1), pp. 95–103. [CrossRef]
Jiang, X., and Mahadevan, S., 2007, “Bayesian Risk-Based Decision Method for Model Validation Under Uncertainty,” Reliab. Eng. Syst. Saf., 92(6), pp. 707–718. [CrossRef]
Sankararaman, S., and Mahadevan, S., 2011, “Model Validation Under Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 96, pp. 1232–1241. [CrossRef]
Bernardo, J. M., Smith, A. F. M., and Berliner, M., 2000, Bayesian Theory, Wiley, New York, New York, USA.
Messer, M., Panchal, J. H., Krishnamurthy, V., Klein, B., Douglas Yoder, P., Allen, J. K., and Mistree, F., 2010, “Model Selection Under Limited Information Using a Value-of-Information-Based Indicator,” ASME J. Mech. Des., 132(12), p. 121008. [CrossRef]
Radhakrishnan, R., and McAdams, D. A., 2005, “A Methodology for Model Selection in Engineering Design,” ASME J. Mech. Des., 127(3), 378–387. [CrossRef]
Robinson, T. D., Eldred, M. S., Willcox, K. E., and Haimes, R., 2008, “Surrogate-Based Optimization Using Multifidelity Models With Variable Parameterization and Corrected Space Mapping,” AIAA J., 46(11), pp. 2814–2822. [CrossRef]
Rangavajhala, S., Liang, C., Mahadevan, S., and Hombal, V. K., 2011, “Concurrent Optimization of Mesh Refinement and Design Parameters in Multidisciplinary Design,” J. Aircr., 49, pp. 1786–1795. [CrossRef]
Sugiyama, M., and Ogawa, H., 2001, “Subspace Information Criterion for Model Selection,” Neural Comput., 13(8), pp. 1863–1889. [CrossRef] [PubMed]
Segalman, D. J., 2005, “A Four-Parameter Iwan Model for Lap-Type Joints,” ASME J. Appl. Mech., 72, pp. 752–760. [CrossRef]
Smallwood, D. O., Gregory, D. L., and Coleman, R. G., 2001, “A Three Parameter Constitutive Model for a Joint Which Exhibits a Power Law Relationship Between Energy Loss and Relative Displacement,” Proceedings of the 72nd Shock and Vibration Symposium.
Sarkar, S., 2009, “Probabilistic Durability Analysis of Cementitious Materials Under External Sulfate Attack,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
Viana, F. A. C., and Haftka, R.T., 2008, “Using Multiple Surrogates for Metamodeling,” Proceedings of the 7th ASMO-UK/ISSMO International Conference on Engineering Design Optimization.
Zhao, D., and Xue, D., 2011, “A Multi-Surrogate Approximation Method for Metamodeling,” Eng. Comput., 27(2), pp. 139–153. [CrossRef]
Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R., and Kevin Tucker, P., 2005, “Surrogate-Based Analysis and Optimization,” Prog. Aerosp. Sci., 41(1), pp. 1–28. [CrossRef]
Villanueva, D., Le Riche, R., Picard, G., and Haftka, R. T., 2012, “Surrogate-Based Agents for Constrained Optimization,” 14th AIAA Non-Deterministic Approaches Conference, Honolulu, AIAA Paper No. 2012-1935.
Rasmussen, C. E., and Williams, C. K. I., 2006, Gaussian Processes for Machine Learning, Springer, New York.
Beck, J. L., and Yuen, K. V., 2004, “Model Selection Using Response Measurements: Bayesian Probabilistic Approach,” J. Eng. Mech., 130, pp. 192–203. [CrossRef]
Rebba, R., and Mahadevan, S., 2006, “Validation of Models With Multivariate Output,” Reliab. Eng. Syst. Saf., 91(8), pp. 861–871. [CrossRef]
Shao, J., 1993, “Linear Model Selection by Cross-Validation,” J. Am. Stat. Assoc., 88, pp. 486–494. [CrossRef]
McFarland, J. M., 2008, “Uncertainty Analysis for Computer Simulations Through Validation and Calibration,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P., 1989, “Design and Analysis of Computer Experiments,” Stat. Sci., 4(4), pp. 409–435. [CrossRef]
MacKay, D. J. C., 1998, “Introduction to Gaussian Processes,” NATO ASI Series, Ser. F, 168, pp. 133–166.
Paciorek, C. J., 2003, “Nonstationary Gaussian Processes for Regression and Spatial Modelling,” Ph.D. thesis, Carnegie Mellon University, Pittsburg, PA.
Goldberg, P. W., Williams, C. K. I., and Bishop, C. M., 1998, “Regression With Input-Dependent Noise: A Gaussian Process Treatment,” Adv. Neural Inf. Process. Syst., 10, pp. 493–499.
Kersting, K., Plagemann, C., Pfaff, P., and Burgard, W., 2007, “Most Likely Heteroscedastic Gaussian Process Regression,” Proceedings of the 24th International Conference on Machine Learning, ACM, pp. 393–400.
Iwan, W. D., 1966, “A Distributed-Element Model for Hysteresis and Its Steady-State Dynamic Response,” ASME J. Appl. Mech., 33, pp. 893–900. [CrossRef]
Segalman, D. J., 2002, “A Four-Parameter Iwan Model for Lap-Type Joints,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND2002-3828.
Lilliefors, H. W., 1967, “On the Kolmogorov-Smirnov Test for Normality With Mean and Variance Unknown,” J. Am. Stat. Assoc., 62(318), pp. 399–402. [CrossRef]
Anderson, T. L., 2005, Fracture Mechanics: Fundamentals and Applications, CRC Press, Boca Raton.
Paris, P. C., and Erdogan, F., 1963, “A Critical Analysis of Crack Propagation Laws,” J. Basic Eng., 85(4), pp. 528–534. [CrossRef]
Forman, R. G., Kearney, V. E., and Engle, R. M., 1967, “Numerical Analysis of Crack Propagation in Cyclic-Loaded Structures,” J. Basic Eng., 89(3), pp. 459–463. [CrossRef]
Klesnil, M., and Lukáš, P., 1972, “Influence of Strength and Stress History on Growth and Stabilisation of Fatigue Cracks,” Eng. Fract. Mech., 4(1), pp. 77–92. [CrossRef]
Shantz, C., 2010, “Uncertainty Quantification in Fatigue Crack Propagation Under Multiaxial Variable Amplitude Loading,” Ph.D. thesis, Vanderbilt University, Nashville, TN.
Sankararaman, S., Ling, Y., Shantz, C., and Mahadevan, S., 2011, “Inference of Equivalent Initial Flaw Size Under Multiple Sources of Uncertainty,” Int. J. Fatigue, 33(2), pp. 75–89. [CrossRef]
Fatemi, A., and Shamsaei, N., 2011, “Multiaxial Fatigue: An overview and Some Approximation Models for Life Estimation,” Elsv., Int. J. Fatigue, 33, pp. 948–958. [CrossRef]
Forth, S. C., Favrow, L. H., Keat, W. D., and Newman, J. A., 2003, “Three-Dimensional Mixed-Mode Fatigue Crack Growth in a Functionally Graded Titanium Alloy,” Eng. Fract. Mech., 70(15), pp. 2175–2185. [CrossRef]
Reis, L., Li, B., and de Freitas, M., 2009, “Crack Initiation and Growth Path Under Multiaxial Fatigue Loading in Structural Steels,” Int. J. Fatigue, 31(11–12), pp. 1660–1668. [CrossRef]
Carter, B. J., Wawrzynek, P. A., and Ingraffea, A. R., 2000, “Automated 3-D Crack Growth Simulation,” Int. J. Numer. Methods Eng., 47, pp. 229–253. [CrossRef]
Maligno, A. R., Rajaratnam, S., Leen, S. B., and Williams, E. J., 2010, “A Three-Dimensional (3d) Numerical Study Of Fatigue Crack Growth Using Remeshing Techniques,” Eng. Fract. Mech., 77(1), pp. 94–111. [CrossRef]
Ingrafea, T., et al. ., 1996, FRANC3D, 3D Fracture Analysis Code, The Cornell University Fracture Group, Cornell University, Ithaca, NY.
Jiang, X., and Mahadevan, S., 2006, “Bayesian Cross-Entropy Methodology for Optimal Design of Validation Experiments,” Meas. Sci. Technol., 17, pp. 1895–1908. [CrossRef]


Grahic Jump Location
Fig. 1

Classification-based model selection. Assigning each point in the domain to the best performing model results in a partition of the domain. The contour represents the classification boundary that partitions the domain. The region in the domain where M1 yields better predictions is marked by circles. The region corresponding to M2 is marked by +.

Grahic Jump Location
Fig. 2

Partition of the domain using Pr[Δe∧12>0] for the 2D problem. Model 1 is selected in regions where Pr[Δe∧12>0]=1. Regions where Pr[Δe∧12>0]=0, model 2 is selected.

Grahic Jump Location
Fig. 3

Effect of model error estimation uncertainties on the selection criterion

Grahic Jump Location
Fig. 4

Model error estimates for 1D example - Stationary Uncertainty

Grahic Jump Location
Fig. 5

Model prediction and prediction uncertainties for 1D example

Grahic Jump Location
Fig. 6

Model error estimates for 1D example - Non-Stationary Uncertainty

Grahic Jump Location
Fig. 7

Classification boundaries and probabilities for 1D example

Grahic Jump Location
Fig. 8

Blurring of the classification boundary due to sparsity of validation points (2D example function)

Grahic Jump Location
Fig. 9

Misclassification rate as a function of number of samples. Two different probability levels α (Eq. (7)) are considered.

Grahic Jump Location
Fig. 10

Misclassification rate as a function of measurement noise. Two different α levels are considered. x axis shows the log(σn2).

Grahic Jump Location
Fig. 11

Iwan model. The Iwan model is a parallel system of spring-slider elements, each with a spring constant k and slider strength φi. The displacement of each element is xi and the displacement of the entire system is u.

Grahic Jump Location
Fig. 12

Smallwood model. Force versus displacement curve, when the applied force is a harmonic function.

Grahic Jump Location
Fig. 13

Iwan and Smallwood models. Data and model prediction errors (Units: F0: lb, Da, Error: in. lb).

Grahic Jump Location
Fig. 14

Iwan and Smallwood models with nonstationary measurement errors (Units: F0: lb, e∧: in. lb)

Grahic Jump Location
Fig. 15

Iwan and Smallwood models: Effect of sparseness of data (Units: F0: lb, e∧: in. lb)

Grahic Jump Location
Fig. 16

Crack growth behavior in metals

Grahic Jump Location
Fig. 17

Model prediction errors (Units: ΔK—Ksi in0.5, errors—in.)

Grahic Jump Location
Fig. 18

Crack growth models. In panel A, bands around each error estimate correspond to ±5σ (Units: ΔK—Ksi in0.5, errors—in.).

Grahic Jump Location
Fig. 19

Finite element meshes of the full model, submodel, and initial crack front




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In