Research Papers: Design Automation

Robustness Metrics: Consolidating the Multiple Approaches to Quantify Robustness

[+] Author and Article Information
Simon Moritz Göhler

Department of Mechanical Engineering,
Technical University of Denmark,
Building 426,
Kongens Lyngby 2800, Denmark
e-mail: simogo@mek.dtu.dk

Tobias Eifler

Department of Mechanical Engineering,
Technical University of Denmark,
Building 426,
Kongens Lyngby 2800, Denmark
e-mail: tobeif@mek.dtu.dk

Thomas J. Howard

Department of Mechanical Engineering,
Technical University of Denmark,
Produktionstorvet, Building 426,
Kongens Lyngby 2800, Denmark
e-mail: thow@mek.dtu.dk

1Corresponding author.

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received February 29, 2016; final manuscript received July 7, 2016; published online September 12, 2016. Assoc. Editor: Zissimos P. Mourelatos.

J. Mech. Des 138(11), 111407 (Sep 12, 2016) (12 pages) Paper No: MD-16-1166; doi: 10.1115/1.4034112 History: Received February 29, 2016; Revised July 07, 2016

The robustness of a design has a major influence on how much the product's performance will vary and is of great concern to design, quality, and production engineers. While variability is always central to the definition of robustness, the concept does contain ambiguity, and although subtle, this ambiguity can have significant influence on the strategies used to combat variability, the way it is quantified and ultimately, the quality of the final design. In this contribution, the literature for robustness metrics was systematically reviewed. From the 108 relevant publications found, 38 metrics were determined to be conceptually different from one another. The metrics were classified by their meaning and interpretation based on the types of the information necessary to calculate the metrics. Four different classes were identified: (1) sensitivity robustness metrics; (2) size of feasible design space robustness metrics; (3) functional expectancy and dispersion robustness metrics; and (4) probability of compliance robustness metrics. The goal was to give a comprehensive overview of robustness metrics and guidance to scholars and practitioners to understand the different types of robustness metrics and to remove the ambiguities of the term robustness. By applying an exemplar metric from each class to a case study, the differences between the classes were further highlighted. These classes form the basis for the definition of four specific subdefinitions of robustness, namely the “robust concept,” “robust design,” “robust function,” and “robust product.”

Copyright © 2016 by ASME
Topics: Robustness , Design
Your Session has timed out. Please sign back in to continue.


Taguchi, G. , Chowdhury, S. , and Wu, Y. , 2005, Taguchi's Quality Engineering Handbook, Wiley, Hoboken, NJ.
Phadke, M. S. , 1989, Quality Engineering Using Robust Design, Prentice Hall, Englewood Cliffs, NJ.
Ebro, M. , Howard, T. J. , and Rasmussen, J. J. , 2012, “ The Foundation for Robust Design: Enabling Robustness Through Kinematic Design and Design Clarity,” International Design Conference, DESIGN, Vol. DS 70, pp. 817–826. https://www.designsociety.org/publication/32050/the_foundation_for_robust_design_enabling_robustness_through_kinematic_design_and_design_clarity
Gremyr, I. , Arvidsson, M. , and Johansson, P. , 2003, “ Robust Design Methodology: Status in the Swedish Manufacturing Industry,” Qual. Reliab. Eng. Int., 19(4), pp. 285–293. [CrossRef]
Thornton, A. C. , Donnelly, S. , and Ertan, B. , 2000, “ More Than Just Robust Design: Why Product Development Organizations Still Contend With Variation and Its Impact on Quality,” Res. Eng. Des., 12(3), pp. 127–143. [CrossRef]
Krogstie, L. , Ebro, M. , and Howard, T. J. , 2014, “ How to Implement and Apply Robust Design: Insights From Industrial Practice,” Total Qual. Manag. Bus. Excellence, 26(11–12), pp. 1–19.
Göhler, S. M. , Ebro, M. , and Howard, T. J. , 2016, “ Mechanisms and Coherences of Robust Design Methodology: A Robust Design Process Proposal,” Total Qual. Manage. Bus. Excellence, 3363, pp. 1–21. [CrossRef]
Beyer, H.-G. , and Sendhoff, B. , 2007, “ Robust Optimization—A Comprehensive Survey,” Comput. Methods Appl. Mech. Eng., 196(33–34), pp. 3190–3218. [CrossRef]
Saltelli, A. , Ratto, M. , Andres, T. , Campolongo, F. , Cariboni, J. , Gatelli, D. , Saisana, M. , and Tarantola, S. , 2008, Global Sensitivity Analysis: The Primer, Wiley, Chichester, UK.
Frey, C. , Patil, S. R. , Frey, C. H. , and Patil, S. R. , 2002, “ Identification and Review of Sensitivity Analysis Methods,” Risk Anal., 22(3), pp. 553–578. [CrossRef] [PubMed]
Thornton, A. C. , 2004, Variation Risk Management: Focusing Quality Improvements in Product Development and Production, Wiley, Hoboken, NJ.
Fowlkes, W. Y. , and Creveling, C. M. , 1995, Engineering Methods for Robust Product Design, Addison-Wesley, Reading, MA.
Eifler, T. , Mathias, J. , Engelhardt, R. , Kloberdanz, H. , Bohn, A. , and Birkhofer, H. , 2011, “ Evaluation of Solution Variants in Conceptual Design by Means of Adequate Sensitivity Indices,” ICED 11— 18th International Conference on Engineering Design—Impacting Society Through Engineering Design, Vol. 9, pp. 314–323. https://www.designsociety.org/publication/30809/evaluation_of_solution_variants_in_conceptual_design_by_means_of_adequate_sensitivity_indices
Zang, C. , Friswell, M. I. , and Mottershead, J. E. , 2005, “ A Review of Robust Optimal Design and Its Application in Dynamics,” Comput. Struct., 83(4–5), pp. 315–326. [CrossRef]
Murphy, T. E. , Tsui, K.-L. , and Allen, J. K. , 2004, “ A Review of Robust Design Methods for Multiple Responses,” Res. Eng. Des., 15(4), pp. 201–215. [CrossRef]
Robinson, T. J. , Borror, C. M. , and Myers, R. H. , 2004, “ Robust Parameter Design: A Review,” Qual. Reliab. Eng. Int., 20(1), pp. 81–101. [CrossRef]
Biolchini, J. , Mian, P. G. , Candida, A. , and Natali, C. , 2005, “ Systematic Review in Software Engineering,” System Engineering and Computer Science Department COPPE/UFRJ, Technical Report ES, 679(05), 45.
Hamby, D. , 1995, “ A Comparison of Sensitivity Analysis Techniques,” Health Phys., 68(2), pp. 195–204. [CrossRef] [PubMed]
Helton, J. C. , Johnson, J. D. , Sallaberry, C. J. , and Storlie, C. B. , 2006, “ Survey of Sampling-Based Methods for Uncertainty and Sensitivity Analysis,” Reliab. Eng. Syst. Saf., 91(10–11), pp. 1175–1209. [CrossRef]
Hutcheson, R. S. , and Mcadams, D. A. , 2012, “ Sensitivity Measures for Use During Conceptual Design,” Int. J. Des. Eng., 5(1), pp. 1–20.
Chen, W. , Allen, J. K. , Tsui, K.-L. , and Mistree, F. , 1996, “ A Procedure for Robust Design: Minimizing Variations Caused by Noise Factors and Control Factors,” ASME J. Mech. Des., 118(4), pp. 478–485. [CrossRef]
Allen, J. K. , Seepersad, C. , Choi, H. , and Mistree, F. , 2006, “ Robust Design for Multiscale and Multidisciplinary Applications,” ASME J. Mech. Des., 128(4), pp. 832–843. [CrossRef]
Dantan, J. Y. , Gayton, N. , Qureshi, A. J. , Lemaire, M. , and Etienne, A. , 2013, “ Tolerance Analysis Approach Based on the Classification of Uncertainty (Aleatory/Epistemic),” Procedia CIRP, 10, pp. 287–293. [CrossRef]
Suh, N. P. , 2001, Axiomatic Design: Advances and Applications, Oxford University Press, New York.
Howard, T. J. , Ebro, M. , Eifler, T. , Petersen, S. , Göhler, S. M. , Christiansen, A. , and Rafn, A. , 2014, “ The Variation Management Framework (VMF) for Robust Design,” 1st International Symposium on Robust Design, Kobenhavn, Denmark, Aug. 14–15. http://orbit.dtu.dk/fedora/objects/orbit:134955/datastreams/file_13747761-ba04-4c70-807b-d6de0d463e74/content
Dai, Z. , Scott, M. J. , and Mourelatos, Z. P. , 2003, “ Incorporating Epistemic Uncertainty in Robust Design,” ASME Paper No. DETC2003/DAC-48713.
Hu, Z. , Du, X. , Kolekar, N. S. , and Banerjee, A. , 2013, “ Robust Design With Imprecise Random Variables and Its Application in Hydrokinetic Turbine Optimization,” Eng. Optim., 46(3), pp. 393–419. [CrossRef]
The Guardian, 2010, “ Toyota's Sticky Accelerator Problem,” accessed Feb. 23, 2016,
Mahalanobis, P. C. , 1936, “ On the Generalised Distance in Statistics,” Proc. Natl. Inst. Sci. India, 12(1), pp. pp. 49–55.
Frey, D. D. , Jahangir, E. , and Engelhardt, F. , 2000, “ Computing the Information Content of Decoupled Designs,” Res. Eng. Des.—Theory, Appl. Concurrent Eng., 12(2), pp. 90–102.
Lasserre, J. B. , 1983, “ An Analytical Expression and an Algorithm for the Volume of a Convex Polyhedron in Rn,” J. Optim. Theory Appl., 39(3), pp. 363–377. [CrossRef]
Hamby, D. M. , 1994, “ A Review of Techniques for Parameter Sensitivity Analysis of Environmental Models,” Environ. Monit. Assess., 32(2), pp. 135–154. [CrossRef] [PubMed]
Minto, B. , 2009, The Pyramid Principle: Logic in Writing and Thinking, Pearson Education, London.
Pahl, G. , Beitz, W. , Feldhusen, J. , and Grote, K.-H. , 2007, Engineering Design: A Systematic Approach, Springer Science & Business Media, London.
Matthiassen, B. , 1997, “ Design for Robustness and Reliability: Improving the Quality Consciousness in Engineering Design,” Technical University of Denmark, Kongens Lyngby, Denmark.
Chakhunashvili, A. , Johansson, P. , and Bergman, B. , 2004, “ Variation Mode and Effect Analysis,” Reliability and Maintainability, 2004 Annual Symposium-RAMS, IEEE, Jan. 26–29, pp. 364–369.
Johannesson, P. , Bergman, B. , Svensson, T. , Arvidsson, M. , Lönnqvist, Å. , Barone, S. , and De Maré, J. , 2013, “ A Robustness Approach to Reliability,” Qual. Reliab. Eng. Int., 29(1), pp. 17–32. [CrossRef]
Ebro, M. , and Howard, T. J. , 2016, “ Robust Design Principles for Reducing Variation in Functional Performance,” J. Eng. Des., pp. 1–18.
Göhler, S. M. , and Howard, T. J. , 2015, “ The Contradiction Index—A New Metric Combining System Complexity and Robustness for Early Design Stages,” ASME Paper No. DETC2015-47255.
Charzyńska, A. , Nalȩcz, A. , Rybiński, M. , and Gambin, A. , 2012, “ Sensitivity Analysis of Mathematical Models of Signaling Pathways,” Biotechnologia, 93(3), pp. 291–308. [CrossRef]
Han, J. S. , and Kwak, B. M. , 2004, “ Robust Optimization Using a Gradient Index: MEMS Applications,” Struct. Multidiscip. Optim., 27(6), pp. 469–478. [CrossRef]
Bhattacharjya, S. , and Chakraborty, S. , 2011, “ Improved Robust Design Optimisation of Structures,” Proc. ICE—Eng. Comput. Mech., 164(1), pp. 47–57.
Sobol, I . M. , 2001, “ Global Sensitivity Indices for Nonlinear Mathematical Models and Their Monte Carlo Estimates,” Math. Comput. Simul., 55(1–3), pp. 271–280. [CrossRef]
Cukier, R. I. , Levine, H. B. , and Shuler, K. E. , 1978, “ Nonlinear Sensitivity Analysis of Multiparameter Model Systems,” J. Comput. Phys., 26(1), pp. 1–42. [CrossRef]
Manache, G. , and Melching, C. S. , 2008, “ Identification of Reliable Regression- and Correlation-Based Sensitivity Measures for Importance Ranking of Water-Quality Model Parameters,” Environ. Model. Software, 23(5), pp. 549–562. [CrossRef]
Lee, M. C. W. , Kelly, D. W. , Degenhardt, R. , and Thomson, R. S. , 2010, “ A Study on the Robustness of Two Stiffened Composite Fuselage Panels,” Compos. Struct., 92(2), pp. 223–232. [CrossRef]
Kwag, S. , and Ok, S.-Y. , 2013, “ Robust Design of Seismic Isolation System Using Constrained Multi-Objective Optimization Technique,” KSCE J. Civ. Eng., 17(5), pp. 1051–1063. [CrossRef]
Caro, S. , Bennis, F. , and Wenger, P. , 2005, “ Tolerance Synthesis of Mechanisms: A Robust Design Approach,” ASME J. Mech. Des., 127(1), pp. 86–94. [CrossRef]
Caro, S. , Bennis, F. , and Wenger, P. , 2005, “ Comparison of Robustness Indices and Introduction of a Tolerance Synthesis Method for Mechanisms,” Can. Congr. Appl. Mech., pp. 3–4. https://hal.archives-ouvertes.fr/hal-00465501/
Ting, K.-L. L. , and Long, Y. , 1996, “ Performance Quality and Tolerance Sensitivity of Mechanisms,” ASME J. Mech. Des., 118(1), pp. 144–150. [CrossRef]
Li, M. , Azarm, S. , and Boyars, A. , 2006, “ A New Deterministic Approach Using Sensitivity Region Measures for Multi-Objective Robust and Feasibility Robust Design Optimization,” ASME J. Mech. Des., 128(4), pp. 874–883. [CrossRef]
Eslamnour, B. , and Ali, S. , 2009, “ Measuring Robustness of Computing Systems,” Simul. Model. Pract. Theory, 17(9), pp. 1457–1467. [CrossRef]
Gunawan, S. , and Azarm, S. , 2004, “ Non-Gradient Based Parameter Sensitivity Estimation for Single Objective Robust Design Optimization,” ASME J. Mech. Des., 126(3), pp. 395–402. [CrossRef]
Kang, Z. , and Bai, S. , 2013, “ On Robust Design Optimization of Truss Structures With Bounded Uncertainties,” Struct. Multidiscip. Optim., 47(5), pp. 699–714. [CrossRef]
Frey, D. D. , 1999, “ Application of Wavelets and Mahalanobis Distances to Robust Design of an Image Classification System,” ASI's 17th Annual Taguchi Methods Symposium, pp. 198–209.
Zhu, J. , and Ting, K.-L. L. , 2001, “ Performance Distribution Analysis and Robust Design,” ASME J. Mech. Des., 123(1), pp. 11–17. [CrossRef]
Lamberti, P. , and Tucci, V. , 2007, “ Interval Approach to Robust Design,” COMPEL Int. J. Comput. Math. Electr. Electron. Eng., 26(20, pp. 280–292.
Du, X. , Sudjianto, A. , and Chen, W. , 2004, “ An Integrated Framework for Optimization Under Uncertainty Using Inverse Reliability Strategy,” ASME J. Mech. Des., 126(4), pp. 562–570. [CrossRef]
Ren, X. , and Rahman, S. , 2013, “ Robust Design Optimization by Polynomial Dimensional Decomposition,” Struct. Multidiscip. Optim., 48(1), pp. 127–148. [CrossRef]
Chen, W. , and Yuan, C. , 1999, “ A Probabilistic-Based Design Model for Achieving Flexibility in Design,” ASME J. Mech. Des., 121(1), pp. 77–83. [CrossRef]
Jin, Y. , and Sendhoff, B. , 2003, “ Trade-Off Between Performance and Robustness: An Evolutionary Multiobjective Approach,” International Conference on Evolutionary Multi-Criterion Optimization, Springer Berlin, Heidelberg, pp. 237–251.
Cheng, Q. , Xiao, C. , Zhang, G. , Gu, P. , and Cai, L. , 2013, “ An Analytical Robust Design Optimization Methodology Based on Axiomatic Design Principles,” Qual. Reliab. Eng. Int., 30(7), pp. 1059–1073. [CrossRef]
Otto, N. , and Antonsson, E. K. , 1994, “ Design Parameter Selection in the Presence of Noise,” Res. Eng. Des., 6(4), pp. 234–246. [CrossRef]
Du, X. , 2012, “ Toward Time-Dependent Robustness Metrics,” ASME J. Mech. Des., 134(1), p. 011004. [CrossRef]
Chen, W. , Simpson, T. W. , Allen, J. K. , and Mistree, F. , 1999, “ Satisfying Ranged Sets of Design Requirements Using Design Capability Indices as Metrics,” Eng. Optim., 31(5), pp. 615–639. [CrossRef]
Heidari, A. , Yoon, Y.-J. , Son, H. , and Choi, H.-J. , 2012, “ Simulation Based Design of Disk Resonator Biosensors Under Fabrication Uncertainty,” ASME J. Mech. Des., 134(4), p. 041005. [CrossRef]


Grahic Jump Location
Fig. 1

Robust Design framework

Grahic Jump Location
Fig. 2

Schematic diagram of the Toyota gas pedal [28] (Courtesy of Guardian News & Media, Ltd.)

Grahic Jump Location
Fig. 3

Size of feasible design space robustness measure (1D)

Grahic Jump Location
Fig. 4

Example for a feasible design space in 2D



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In