Research Papers: Design Theory and Methodology

An Experimental Study on the Influence That Failure Number, Specialization, and Controls Have on Confidence in Predicting System Failures1

[+] Author and Article Information
Somaiah Thimmaiah

Mechanical Engineering,
Clemson University,
Clemson, SC 29634-0921
e-mail: sthimma@clemson.edu

Keith Phelan

Mechanical Engineering,
Clemson University,
Clemson, SC 29634-0921
e-mail: ktphela@clemson.edu

Joshua D. Summers

Mechanical Engineering,
Clemson University,
Clemson, SC 29634-0921
e-mail: jsummer@clemson.edu

2Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received June 18, 2015; final manuscript received August 31, 2016; published online November 11, 2016. Assoc. Editor: Irem Tumer.

J. Mech. Des 139(1), 011102 (Nov 11, 2016) (12 pages) Paper No: MD-15-1426; doi: 10.1115/1.4034789 History: Received June 18, 2015; Revised August 31, 2016

Design reviews are typically used for three types of design activities: (1) identifying errors, (2) assessing the impact of the errors, and (3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Nondomain generalists (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to provide their own opinion or confidence on whether the system would still successfully achieve the stated objectives. The confidence level for all domain populations decreased significantly as the number of design errors increased (largest p-value = 0.0793), and this decrease in confidence is more significant as the number of design errors increases. The impact on confidence is lower when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334) as the confidence decreased faster for domain general engineers as compared to domain specialists (p = < 0.0001). The domain specialists showed higher confidence in making decisions than domain generalists and nondomain generalists as the design errors increase.

Copyright © 2017 by ASME
Topics: Design , Errors , Failure , Testing
Your Session has timed out. Please sign back in to continue.


Schmidt, R. , 2010, “ Risk Assessment in Research Projects,” Annual Reliability and Maintainability Symposium (RAMS), IEEE, San Jose, CA, pp. 1–6.
Clarkson, P. J. , Simons, C. , and Eckert, C. , 2004, “ Predicting Change Propagation in Complex Design,” ASME J. Mech. Des., 126(5), pp. 788–797. [CrossRef]
Shankar, P. , Morkos, B. W. , and Summers, J. D. , 2012, “ Reasons for Change Propagation: A Case Study in an Automotive OEM,” Res. Eng. Des., 23(4), pp. 291–303. [CrossRef]
Shankar, P. , Summers, J. , and Phelan, K. , 2014, “ A Verification and Validation Planning Method to Address Propagation Effects in Engineering Design,” International Symposium on Tools and Methods of Competitive Engineering, Budapest, Hungary, pp. 635–648.
Pfeiffer, T. , 2002, Quality Management: Strategies, Methods, Techniques, Hanser Fachbuchverlag, Munich, Germany.
Hawkins, P. G. , and Woollons, D. J. , 1998, “ Failure Modes and Effects Analysis of Complex Engineering Systems Using Functional Models,” Artif. Intell. Eng., 12(4), pp. 375–397. [CrossRef]
Cotnareanu, T. , 1999, “ Old Tools–New Uses: Equipment FMEA—A Tool for Preventive Maintenance,” Qual. Prog., 32(12), pp. 48–52.
Haik, Y. , and Shahin, T. , 2010, Engineering Design Process, Cengage Learning, Stamford, CT.
Dieter, G. E. , and Schmidt, L. C. , 2009, Engineering Design, McGraw-Hill Higher Education, New York.
Shimizu, H. , Imagawa, T. , and Noguchi, H. , 2003, “ Reliability Problem Prevention Method for Automotive Components—Development of GD3 Activity and DRBFM (Design Review Based on Failure Mode),” SAE Technical Paper No. 2003-01-2877.
Shimizu, H. , Otsuka, Y. , and Noguchi, H. , 2010, “ Design Review Based on Failure Mode to Visualise Reliability Problems in the Development Stage of Mechanical Products,” Int. J. Veh. Des., 53(3), pp. 149–165. [CrossRef]
Schmidt, R. , Riedel, G. J. , and Kangas, K. , 2011, “ Risk Assessment Using Design Review Based on Failure Mode,” Annual Reliability and Maintainability Symposium (RAMS), IEEE, New York, Jan. 24–27.
Hannah, R. , Joshi, S. , and Summers, J. D. , 2011, “ A User Study of Interpretability of Engineering Design Representations,” J. Eng. Des., 23(6), pp. 443–468. [CrossRef]
Sulistyawati, K. , Wickens, C. D. , and Chui, Y. P. , 2011, “ Prediction in Situation Awareness: Confidence Bias and Underlying Cognitive Abilities,” Int. J. Aviat. Psychol., 21(2), pp. 153–174. [CrossRef]
Lin, S. W. , and Bier, V . M. , 2008, “ A Study of Expert Overconfidence,” Reliab. Eng. Syst. Saf., 93(2008), pp. 711–721. [CrossRef]
Bier, V. , 2004, “ Implications of the Research on Expert Overconfidence and Dependence,” Reliab. Eng. Syst. Saf., 85, pp. 321–329. [CrossRef]
Pahl, G. , Beitz, W. , Wallace, K. , and Blessing, L. , 2007, Engineering Design: A Systematic Approach, Springer-Verlag, London.
Ostergaard, K. J. , Wetmore, W. R., III , Divekar, A. , Vitali, H. , and Summers, J. D. , 2005, “ An Experimental Methodology for Investigating Communication in Collaborative Design Review Meetings,” CoDesign, 1(3), pp. 169–185. [CrossRef]
Sater-Black, K. , and Iverson, N. , 1994, “ How to Conduct a Design Review,” Mech. Eng., 116(3), pp. 89–93.
Wetmore, W. R. , and Summers, J. D. , 2003, “ Group Decision Making: Friend or Foe?,” Engineering Management Conference, IEMC’03, Managing Technologically Driven Organizations: The Human Side of Innovation and Change, Nov. 2–4, pp. 405–409.
Wetmore, W. R., III , Summers, J. D. , and Greenstein, J. S. , 2010, “ Experimental Study of Influence of Group Familiarity and Information Sharing on Design Review Effectiveness,” J. Eng. Des., 21(1), pp. 111–126. [CrossRef]
Collins, S. T. , Yassine, A. A. , and Borgatti, S. P. , 2009, “ Evaluating Product Development Systems Using Network Analysis,” Syst. Eng., 12(1), pp. 55–68. [CrossRef]
Hammond, J. , Lafayette, W. , Koubek, R. J. , and Harvey, C. M. , 2001, “ Distributed Collaboration for Engineering Design: A Review and Reappraisal,” Hum. Factors Ergon. Manuf., 11(1), pp. 35–52. [CrossRef]
Pugh, S. , 1991, Total Design: Integrated Methods for Successful Product Engineering, Addison-Wesley Publishing Company, Wokingham, UK.
Hilts, S. , Johnson, K. , and Turoff, M. , 1986, “ Experiments in Group Decision-Making: Communication Process and Outcome in Face-to-Face Versus Computerized Conferences,” Hum. Commun. Res., 13(2), pp. 225–252. [CrossRef]
DoD, 1980, “ Procedures for Performing a Failure Mode, Effects and Criticality Analysis,” U.S. Department of Defense, Washington, DC, Standard No. MIL-STD-1629A.
Teoh, P. C. , and Case, K. , 2005, “ An Evaluation of Failure Modes and Effects Analysis Generation Method for Conceptual Design,” Int. J. Comput. Integr. Manuf., 18(4), pp. 279–293. [CrossRef]
Xu, K. , Tang, L. C. , Xie, M. , Ho, S. L. , and Zhu, M. L. , 2002, “ Fuzzy Assessment of FMEA for Engine Systems,” Reliab. Eng. Syst. Saf., 75(1), pp. 17–29. [CrossRef]
Dinsmore, D. L. , and Parkinson, M. M. , 2012, “ What are Confidence Judgments Made of? Students' Explanations for Their Confidence Ratings and What That Means for Calibration,” Learn. Instr., 24, pp. 1–11.
Thorndike, E. L. , and Gates, A. I. , 1929, Elementary Principles of Education, Macmillan Company, New York.
Pedhazur, E. J. , 1997, Multiple Regression in Behavioral Research, Harcourt Brace, Orlando, FL.
Finney, S. J. , and DiStefano, C. , 2006, Non-Normal and Categorical Data in Structural Equation Modeling, Information Age Publishing, Greenwich, CT.
Schraw, G. , Potenza, M. T. , and Nebelsick-Gullet, L. , 1993, “ Constraints on the Calibration of Performance,” Contemp. Educ. Psychol., 18(4), pp. 455–463. [CrossRef]
Bernstein, E. M. , and Putnam, F. W. , 1986, “ Development, Reliability, and Validity of a Dissociation Scale,” J. Nerv. Ment. Dis., 174(12), pp. 727–735. [CrossRef] [PubMed]
Aitken, R. C. , 1969, “ Measurement of Feelings Using Visual Analogue Scales,” Proc. R. Soc. Med., 62(10), pp. 989–993. [PubMed]
Miller, G. A. , 1956, “ The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information,” Psychol. Rev., 63(2), pp. 81–97. [CrossRef] [PubMed]
Cowan, N. , 2001, “ The Magical Number 4 in Short-Term Memory: A Reconsideration of Mental Storage Capacity,” Behav. Brain Sci., 24(1), pp. 87–114. [CrossRef] [PubMed]
Farrington, J. , 2011, “ Seven Plus or Minus Two,” Perform. Improv. Q., 23(4), pp. 113–116. [CrossRef]
Thimmaiah, S. , 2013, “ An Experimental Study on the Influence That Failure Number, Specialization, and Domain Have on Confidence in Predicting System Failures,” M.S. thesis, Clemson University, Clemson, SC.


Grahic Jump Location
Fig. 1

Example of FMEA worksheet for a pressurized mud box seal testing system

Grahic Jump Location
Fig. 2

Tent ballast testing (equipment) setup: (a) iconic schematic of test setup and (b) implemented schematic of test setup

Grahic Jump Location
Fig. 3

Assessment of confidence (%) of system success for each design error

Grahic Jump Location
Fig. 4

Example of error/failure mode worksheet (scenario 2)

Grahic Jump Location
Fig. 5

Confidence level comparison between different domain populations

Grahic Jump Location
Fig. 6

Histogram of student confidence responses

Grahic Jump Location
Fig. 7

Effect of controls on all domain populations

Grahic Jump Location
Fig. 10

Confidence versus design errors for DS with and without controls

Grahic Jump Location
Fig. 9

Confidence versus design errors for NDG with and without controls

Grahic Jump Location
Fig. 8

Confidence versus design errors for DG with and without controls



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In