Research Papers

Characterizing Uncertainty Attributable to Surrogate Models

[+] Author and Article Information
Jie Zhang

Postdoctoral Research Associate
Multidisciplinary Design and Optimization
Laboratory (MDOL),
Department of Mechanical and
Aerospace Engineering,
Syracuse University,
Syracuse, NY 13244
e-mail: jzhang56@syr.edu

Souma Chowdhury

Assistant Research Professor
Department of Mechanical Engineering,
Department of Electrical and
Computer Engineering,
Mississippi State University,
Mississippi State, MS 39762
e-mail: souma.chowdhury@msstate.edu

Ali Mehmani

Multidisciplinary Design and Optimization
Laboratory (MDOL),
Department of Mechanical and
Aerospace Engineering,
Syracuse University,
Syracuse, NY 13244
e-mail: amehmani@syr.edu

Achille Messac

Earnest W. and Mary Ann Deavenport Jr.
Endowed Chair,
Fellow ASME
Department of Aerospace Engineering,
Bagley College of Engineering,
Mississippi State University,
Mississippi State, MS 39762
e-mail: messac@bagley.msstate.edu

1Present address: National Renewable Energy Laboratory (NREL), Golden, CO 80401.

2Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received December 13, 2012; final manuscript received October 23, 2013; published online January 10, 2014. Assoc. Editor: Irem Y. Tumer.

J. Mech. Des 136(3), 031004 (Jan 10, 2014) (11 pages) Paper No: MD-12-1608; doi: 10.1115/1.4026150 History: Received December 13, 2012; Revised October 23, 2013

This paper investigates the characterization of the uncertainty in the prediction of surrogate models. In the practice of engineering, where predictive models are pervasively used, the knowledge of the level of modeling error in any region of the design space is uniquely helpful for design exploration and model improvement. The lack of methods that can explore the spatial variation of surrogate error levels in a wide variety of surrogates (i.e., model-independent methods) leaves an important gap in our ability to perform design domain exploration. We develop a novel framework, called domain segmentation based on uncertainty in the surrogate (DSUS) to segregate the design domain based on the level of local errors. The errors in the surrogate estimation are classified into physically meaningful classes based on the user's understanding of the system and/or the accuracy requirements for the concerned system analysis. The leave-one-out cross-validation technique is used to quantity the local errors. Support vector machine (SVM) is implemented to determine the boundaries between error classes, and to classify any new design point into the pertinent error class. We also investigate the effectiveness of the leave-one-out cross-validation technique in providing a local error measure, through comparison with actual local errors. The utility of the DSUS framework is illustrated using two different surrogate modeling methods: (i) the Kriging method and (ii) the adaptive hybrid functions (AHF). The DSUS framework is applied to a series of standard test problems and engineering problems. In these case studies, the DSUS framework is observed to provide reasonable accuracy in classifying the design-space based on error levels. More than 90% of the test points are accurately classified into the appropriate error classes.

Copyright © 2014 by ASME
Your Session has timed out. Please sign back in to continue.


Haldar, A., and Mahadevan, S., 2000, Probability, Reliability, and Statistical Methods in Engineering Design, Wiley, New York.
Picheny, V., 2009, “Improving Accuracy and Compensating for Uncertainty in Surrogate Modeling,” Ph.D. thesis, Aerospace Engineering, University of Florida, Gainesville, FL.
Keane, A. J., and Nair, P. B., 2005, Computational Approaches for Aerospace Design: The Pursuit of Excellence, Wiley, New York.
Myers, R. H., and Montgomery, D. C., 2002, Response Surface Methodology: Process and Product Optimization Using Designed Experiments, 2nd ed., Wiley, New York.
Giunta, A. A., and Watson, L., 1998, “A Comparison of Approximation Modeling Techniques: Polynomial Versus Interpolating Models,” NASA, Technical Report AIAA-98-4758.
Sakata, S., Ashida, F., and Zako, M., 2003, “Structural Optimization Using Kriging Approximation,” Comput. Methods Appl. Mech. Eng., 192(7–8), pp. 923–939. [CrossRef]
Cressie, N., 1993, Statistics for Spatial Data, Wiley, New York.
Hardy, R. L., 1971, “Multiquadric Equations of Topography and Other Irregular Surfaces,” J. Geophys. Res., 76, pp. 1905–1915. [CrossRef]
Jin, R., Chen, W., and Simpson, T., 2001, “Comparative Studies of Metamodelling Techniques Under Multiple Modelling Criteria,” Struct. Multidiscip. Optim., 23(1), pp. 1–13. [CrossRef]
Mullur, A. A., and Messac, A., 2005, “Extended Radial Basis Functions: More Flexible and Effective Metamodeling,” AIAA J., 43(6), pp. 1306–1315. [CrossRef]
Duda, R. O., Hart, P. E., and Stork, D. G., 2000, Pattern Classification, 2nd ed., Wiley, New York.
Yegnanarayana, B., 2004, Artificial Neural Networks, PHI Learning Pvt. Ltd., New Delhi, India.
Clarke, S. M., Griebsch, J. H., and Simpson, T. W., 2005, “Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses,” ASME J. Mech. Des., 127(6), pp. 1077–1087. [CrossRef]
Vapnik, V., 1995, The Nature of Statistical Learning Theory, Springer, New York.
Basudhar, A., and Missoum, S., 2008, “Adaptive Explicit Decision Functions for Probabilistic Design and Optimization Using Support Vector Machines,” Comput. Struct., 86(19–20), pp. 1904–1917. [CrossRef]
Forrester, A. I. J., and Keane, A. J., 2009, “Recent Advances in Surrogate-Based Optimization,” Prog. Aerosp. Sci., 45(1–3), pp. 50–79. [CrossRef]
Queipo, N., Haftka, R., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P., 2005, “Surrogate-Based Analysis and Optimization,” Prog. Aerosp. Sci., 41(1), pp. 1–28. [CrossRef]
Wang, G., and Shan, S., 2007, “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” ASME J. Mech. Des., 129(4), pp. 370–380. [CrossRef]
Simpson, T. W., Toropov, V., Balabanov, V., and Viana, F. A. C., 2008, “Design and Analysis of Computer Experiments in Multidisciplinary Design Optimization: A Review of How Far we Have Come or Not,” 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA.
Zerpa, L. E., Queipo, N. V., Pintos, S., and Salager, J., 2005, “An Optimization Methodology of Alkaline-Urfactant-Polymer Flooding Processes Using Field Scale Numerical Simulation and Multiple Surrogates,” J. Pet. Sci. Eng., 47(3–4), pp. 197–208. [CrossRef]
Goel, T., Haftka, R. T., Shyy, W., and Queipo, N. V., 2007, “Ensemble of Surrogates,” Struct. Multidiscip. Optim., 33(3), pp. 199–216. [CrossRef]
Sanchez, E., Pintos, S., and Queipo, N. V., 2008, “Toward an Optimal Ensemble of Kernel-Based Approximations With Engineering Applications,” Struct. Multidiscip. Optim., 36(3), pp. 247–261. [CrossRef]
Acar, E., and Rais-Rohani, M., 2009, “Ensemble of Metamodels With Optimized Weight Factors,” Struct. Multidiscip. Optim., 37(3), pp. 279–294. [CrossRef]
Viana, F. A. C., Haftka, R. T., and Steffen, V., 2009, “Multiple Surrogates: How Cross-Validation Errors Can Help us to Obtain the Best Predictor,” Struct. Multidiscip. Optim., 39(4), pp. 439–457. [CrossRef]
Apley, D. W., Liu, J., and Chen, W., 2006, “Understanding the Effects of Model Uncertainty in Robust Design With Computer Experiments,” ASME J. Mech. Des., 128(4), pp. 945–958. [CrossRef]
Kennedy, M. C., and O'Hagan, A., 2001, “Bayesian Calibration of Computer Models,” J. R. Stat. Soc.: Ser. B, 63(3), pp. 425–464. [CrossRef]
Neufeld, D., Behdinan, K., and Chung, J., 2010, “Aircraft Wing Box Optimization Considering Uncertainty in Surrogate Models,” Struct. Multidiscip. Optim., 42(5), pp. 745–753. [CrossRef]
Eldred, M., Giunta, A., Wojtkiewicz, S. F., and Trucano, T., 2002, “Formulations for Surrogate-Based Optimization Under Uncertainty,” 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, AIAA.
Jones, D., Schonlau, M., and Welch, W., 1998, “Efficient Global Optimization of Expensive Black-Box Functions,” J. Global Optim., 13(4), pp. 455–492. [CrossRef]
Viana, F. A. C., and Haftka, R. T., 2009, “Importing Uncertainty Estimates From one Surrogate to Another,” 50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA.
Xiong, Y., Chen, W., and Tsui, K., 2008, “A New Variable-Fidelity Optimization Framework Based on Model Fusion and Objective-Oriented Sequential Sampling,” ASME J. Mech. Des., 130(11), p. 111401. [CrossRef]
Chen, S., Xiong, Y., and Chen, W., 2009, “Multiresponse and Multistage Metamodeling Approach for Design Optimization,” AIAA J., 47(1), pp. 206–218. [CrossRef]
Huang, D., Allen, T. T., Notz, W. I., and Zeng, N., 2006, “Global Optimization of Stochastic Black-Box Systems Via Sequential Kriging Meta-Models,” J. Global Optim., 34(3), pp. 441–466. [CrossRef]
Zhang, J., Chowdhury, S., Zhang, J., Messac, A., and Castillo, L., 2013, “Adaptive Hybrid Surrogate Modeling for Complex Systems,” AIAA J., 51(3), pp. 643–656. [CrossRef]
Zhang, J., Chowdhury, S., Messac, A., and Castillo, L., 2012, “A Response Surface-Based Cost Model for Wind Farm Design,” Energy Policy, 42, pp. 538–550. [CrossRef]
Chowdhury, S., Zhang, J., Messac, A., and Castillo, L., 2012, “Unrestricted Wind Farm Layout Optimization (UWFLO): Investigating Key Factors Influencing the Maximum Power Generation,” Renewable Energy, 38(1), pp. 16–30. [CrossRef]
Chowdhury, S., Zhang, J., Messac, A., and Castillo, L., 2013, “Optimizing the Arrangement and the Selection of Turbines for Wind Farms Subject to Varying Wind Conditions,” Renewable Energy, 52, pp. 273–282. [CrossRef]
Forrester, A. I. J., Sóbester, A., and Keane, A. J., 2008, Engineering Design via Surrogate Modelling: A Practical Guide, Wiley, New York.
Joseph, V. R., Hung, Y., and Sudjianto, A., 2008, “Blind Kriging: A New Method for Developing Metamodels,” ASME J. Mech. Des., 130(3), p. 031102. [CrossRef]
Viana, F. A. C., and Haftka, R. T., 2009, “Cross Validation Can Estimate How Well Prediction Variance Correlates With Error,” AIAA J., 47(9), pp. 2266–2270. [CrossRef]
Viana, F. A. C., Picheny, V., and Haftka, R. T., 2010, “Using Cross Validation to Design Conservative Surrogates,” AIAA J., 48(10), pp. 2286–2298. [CrossRef]
Duan, K., and Keerthi, S. S., 2005, “Which is the Best Multiclass SVM Method? An Empirical Study,” Multiple Classifier Syst., 3541, pp. 732–760.
Hsu, C. W., and Lin, C. J., 2002, “A Comparison of Methods for Multiclass Support Vector Machines,” IEEE Trans. Neural Networks, 13(2), pp. 415–425. [CrossRef]
Chang, C. C., and Lin, C. J., 2011, “LIBSVM: A Library for Support Vector Machines,” ACM Trans. Intell. Syst. Technol., 2(3), pp. 27:1–27:27. [CrossRef]
Zhang, J., Chowdhury, S., and Messac, A., 2012, “An Adaptive Hybrid Surrogate Model,” Struct. Multidiscip. Optim., 46(2), pp. 223–238. [CrossRef]
Audze, P., and Eglais, V., 1997, “New Approach for Planning Out of Experiments,” Prob. Dyn. Strengths, 35, pp. 104–107.
Lophaven, S. N., Nielsen, H. B., and Sondergaard, J., 2002, Dace—A Matlab Kriging Toolbox, Version 2.0,” Technical University of Denmark, Technical Report, Informatics and Mathematical Modelling Report IMM-REP-2002-12.
Zhang, J., Chowdhury, S., Messac, A., and Castillo, L., 2011, “A Comprehensive Measure of the Energy Resource Potential of a Wind Farm Site,” ASME 2011 5th International Conference on Energy Sustainability, ASME.
Chowdhury, S., Messac, A., and Khire, R. A., 2011, “Comprehensive Product Platform Planning (cp3) Framework,” ASME J. Mech. Des., 133(10), p. 101004. [CrossRef]
Zhang, J., Chowdhury, S., Messac, A., Castillo, L., and Lebron, J., 2010, “Response Surface Based Cost Model for Onshore Wind Farms Using Extended Radial Basis Functions,” ASME 2010 International Design Engineering Technical Conferences (IDETC), ASME.
GE, 2010, GE Energy 1.5MW Wind Turbine Brochure, General Electric, http://www.gepower.com/
Chowdhury, S., Messac, A., and Khire, R. A., 2013, “Investigating the Commonality Attributes for Scaling Product Families Using Comprehensive Product Platform Planning (cp3),” Struct. Multidiscip. Optim., 48, pp. 1089–1107.
Goldberg, M., 2009, Jobs and Economic Development Impact (JEDI) Model, National Renewable Energy Laboratory, Golden, CO.
NDSU, 2012, “The North Dakota Agricultural Weather Network,” http://ndawn.ndsu.nodak.edu


Grahic Jump Location
Fig. 1

The illustration of the prediction uncertainty modeling (class 1: low error; class 2: medium error; class 3: high error)

Grahic Jump Location
Fig. 2

The framework of the DSUS methodology

Grahic Jump Location
Fig. 3

Comparison of cross-validation errors and actual local errors (1-variable function): (a) AHF and (b) Kriging

Grahic Jump Location
Fig. 4

Comparison of cross-validation errors and actual local errors (Dixon & Price function): (a) actual local errors (AHF), (b) cross-validation errors (AHF), (c) actual local errors (Kriging), and (d) cross-validation errors (Kriging)




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In