Research Papers: Design Theory and Methodology

An Approach to Study Associations Between Design Concepts and Design Outcomes

[+] Author and Article Information
Shun Takai

Department of Technology,
Northern Illinois University,
DeKalb, IL 60115
e-mail: stakai@niu.edu

Marcos Esterman

Department of Industrial
and Systems Engineering,
Rochester Institute of Technology,
Rochester, NY 14623-5603
e-mail: mxeeie@rit.edu

Ashok Midha

Department of Mechanical
and Aerospace Engineering,
Missouri University of
Science and Technology,
Rolla, MO 65409-0500
e-mail: midha@mst.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received April 22, 2012; final manuscript received August 30, 2014; published online February 6, 2015. Assoc. Editor: Jonathan Cagan.

J. Mech. Des 137(4), 041101 (Apr 01, 2015) (12 pages) Paper No: MD-12-1219; doi: 10.1115/1.4028992 History: Received April 22, 2012; Revised August 30, 2014; Online February 06, 2015

This paper proposes an approach to investigate associations between design concepts and design outcomes. In the proposed approach, three deliverables (concept sketches, proof-of-concept (POC) prototypes, and final products) are evaluated using metrics called creative product semantic scale (CPSS). CPSS scores are analyzed using two methods: correlation analysis and classification tree analysis. Correlation analysis is used to investigate associations between the concept sketches or the POC prototypes and the final products. Classification tree analysis (together with leave-one-out cross-validation (LOOCV) and permutation test) is used to investigate relationships between CPSS scores and the retention of design concepts (i.e., concepts that are embodied in the final products). The proposed approach is illustrated using deliverables in a project-based design class.

Copyright © 2015 by ASME
Your Session has timed out. Please sign back in to continue.


Leary, W. E., 2002, “The Inquiring Minds Behind 200 Years of Inventions,” The New York Times, Oct. 22, 2002.
Brown, T., 2008, “Design Thinking,” Harv. Bus. Rev., 86(6), pp. 84–92. [PubMed]
Åstebro, T., 1998, “Basic Statistics on the Success Rate and Profits for Independent Inventors,” Entrepreneurship Theory Pract., 23(2), pp. 41–48.
Ward, A. C., Liker, J. K., Cristiano, J. J., and Sobek, II, D. K., 1995, “The Second Toyota Paradox: How Delaying Decisions Can Make Better Cars Faster,” Sloan Manage. Rev., 36(3), pp. 43–61.
Fredman, G., 2002, The IDEO Difference, Hemispheres, Pace Communications, Greensboro, NC, pp. 52–57.
Zuber, C. D., Alterescu, V., and Chow, M. P., 2005, “Fail Often to Succeed Sooner: Adventures in Innovation,” Perm. J., 9(4), pp. 44–49. [PubMed]
National Academy of Engineering, 2004, The Engineer of 2020: Visions of Engineering in the New Century, National Academies, Washington, DC.
National Academy of Engineering, 2005, Educating the Engineer of 2020: Adapting Engineering Education to the New Century, National Academies, Washington, DC.
Dym, C. L., Agogino, A. M., Frey, D. D., Eris, O., and Leifer, L. J., 2005, “Engineering Design Thinking, Teaching, and Learning,” J. Eng. Educ., 94(1), pp. 103–120. [CrossRef]
Baillie, C., 2002, “Enhancing Creativity in Engineering Students,” Eng. Sci. Educ. J., 11(5), pp. 185–192. [CrossRef]
Liu, Z., and Schönwetter, D. J., 2004, “Teaching Creativity in Engineering,” Int. J. Eng. Educ., 20(5), pp. 801–808.
Amabile, T. M., 1982, “Social Psychology of Creativity: A Consensual Assessment Technique,” J. Pers. Soc. Psychol., 43(5), pp. 997–1013. [CrossRef]
Campbell, D. T., 1960, “Blind Variation and Selective Retention in Creative Though as in Other Knowledge Process,” Psychol. Rev., 67(6), pp. 380–400. [CrossRef] [PubMed]
Mednick, S. A., 1962, “The Association Basis of the Creative Process,” Psychol. Rev., 69(3), pp. 220–232. [CrossRef] [PubMed]
Rothenberg, A., 1979, The Emerging Goddess: The Creative Process in Art, Science, and Other Fields, University of Chicago, Chicago, IL.
Finke, R. A., Ward, T. B., and Smith, S. M., 1992, Creative Cognition: Theory, Research, and Applications, MIT, Cambridge, MA.
Finke, R. A., 1995, “Creative Realism,” Creative Cognition Approach, S. M.Smith, T. B.Ward, and R. A.Finke, eds., MIT, Cambridge, MA.
Ward, T. B., Smith, S. M., and Vaid, J., 1997, Creative Thought: An Investigation of Conceptual Structures and Processes, American Psychological Association, Washington, DC. [CrossRef]
Ward, T. B., Smith, S. M., and Vaid, J., 1997, “Conceptual Structures and Processes in Creative Thought,” Creative Thought: An Investigation of Conceptual Structures and Processes, T. B.Ward, S. M.Smith, and J.Vaid, eds., American Psychological Association, Washington, DC.
Domino, G., 1970, “Identification of Potentially Creative Persons From the Adjective Check List,” J. Consult. Clin. Psychol., 35(1), pp. 48–51. [CrossRef] [PubMed]
Gough, H. A., 1979, “A Creative Personality Scale for the Adjective Check List,” J. Pers. Soc. Psychol., 37(8), pp. 1398–1407. [CrossRef]
Strack, S., and Lorr, M., 1990, “Item Factor Structure of the Personality Adjective Check List,” J. Pers. Assess., 55(1–2), pp. 86–94. [CrossRef] [PubMed]
Kaduson, H. G., 1991, “Concurrent Validity of the Creative Personality Scale of the Adjective Check List,” Psychol. Rep., 69(2), pp. 601–602. [CrossRef]
Torrance, E. P., 1962, Guiding Creative Talent, Prentice-Hall, Englewood Cliffs, NJ. [CrossRef]
Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Des. Stud., 12(1), pp. 3–11. [CrossRef]
Purcell, A. T., and Gero, J. S., 1996, “Design and Other Types of Fixation,” Des. Stud., 17(4), pp. 363–383. [CrossRef]
Kokotovich, V., and Purcell, T., 2000, “Mental Synthesis and Creativity in Design: An Experimental Examination,” Des. Stud., 21(5), pp. 437–449. [CrossRef]
Hey, J., Linsey, J., Agogino, A., and Wood, K., 2008, “Analogies and Metaphors in Creative Design,” Int. J. Eng. Educ., 24(2), pp. 283–294.
Linsey, J., Wood, K., and Markman, A., 2008, “Modality and Representation in Analogy,” Artif. Intell. Eng. Des. Anal. Manuf., 22(2), pp. 85–100. [CrossRef]
Tseng, I., Moss, J., Cagan, J., and Kotovsky, K., 2008, “The Role of Timing and Analogical Similarity in the Stimulation of Idea Generation in Design,” Des. Stud., 29(3), pp. 203–221. [CrossRef]
Ball, L. J., and Christensen, B. T., 2009, “Analogical Reasoning and Mental Simulation in Design: Two Strategies Linked to Uncertainty Resolution,” Des. Stud., 30(2), pp. 169–186. [CrossRef]
Nagai, Y., Taura, T., and Mukai, F., 2009, “Concept Blending and Dissimilarity: Factors for Creative Concept Generation Process,” Des. Stud., 30(6), pp. 648–675. [CrossRef]
Linsey, J., Tseng, I., Fu, K., Cagan, J., Wood, K., and Schunn, C., 2010, “A Study of Design Fixation, its Mitigation and Perception in Engineering Design Faculty,” ASME J. Mech. Des., 132(4), p. 041003. [CrossRef]
Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., and Markman, A. B., 2011, “An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods,” ASME J. Mech. Des., 133(3), p. 031008. [CrossRef]
Linsey, J., Markman, A., and Wood, K., 2012, “Design by Analogy: A Study of the WordTree Method for Problem Re-Representation,” ASME J. Mech. Des., 134(4), p. 041009. [CrossRef]
Schütze, M., Sachse, P., and Römer, A., 2003, “Support Value of Sketching in the Design Process,” Res. Eng. Des., 14(2), pp. 89–97.
Dong, A., 2004, “A Document Analysis Method for Characterizing Design Team Performance,” ASME J. Mech. Des., 126(3), pp. 378–385. [CrossRef]
Ruder, J. A., and Sobek, II, D. K., 2007, “Experiment on a System-Level Design Tool,” J. Eng. Des., 18(4), pp. 327–342. [CrossRef]
Song, S., and Agogino, A. M., 2004, “Insights on Designers' Sketching Activities in New Product Design Teams,” ASME Paper No. DETC2004-57474. [CrossRef]
Yang, M. C., 2009, “Observations on Concept Generation and Sketching in Engineering Design,” Res. Eng. Des., 20(1), pp. 1–11. [CrossRef]
Wilde, D. J., 2009, Teamology: The Construction and Organization of Effective Teams, Springer, NY.
Shah, J. J., Kulkarni, S. V., and Vargas-Hernandez, N., 2000, “Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments,” ASME J. Mech. Des., 122(4), pp. 377–384. [CrossRef]
Shah, J. J., Vargas-Hernandez, N., and Smith, S. M., 2003, “Metrics for Measuring Ideation Effectiveness,” Des. Stud., 24(2), pp. 111–134. [CrossRef]
Chusilp, P., and Jin, Y., 2006, “Impact of Mental Iteration on Concept Generation,” ASME J. Mech. Des., 128(1), pp. 14–25. [CrossRef]
O'Quin, K., and Besemer, S. P., 1989, “The Development, Reliability, and Validity of the Revised Creative Product Semantic Scale,” Creativity Res. J., 2(4), pp. 267–278. [CrossRef]
Besemer, S. P., and Treffinger, D. J., 1981, “Analysis of Creative Products: Review and Synthesis,” J. Creative Behav., 15(3), pp. 158–178. [CrossRef]
Besemer, S. P., and O'Quin, K., 1986, “Analyzing Creative Products: Refinement and Test of a Judging Instrument,” J. Creative Behav., 20(2), pp. 115–126. [CrossRef]
Besemer, S. P., and O'Quin, K., 1987, “Creative Product Analysis: Testing a Model by Developing a Judging Instrument,” Frontiers of Creativity Research, S. G.Isaksen, ed., Bearly Limited, Buffalo, NY, pp. 341–357.
Dahl, D. W., Chattopadhyay, A., and Gorn, G. J., 2001, “The Importance of Visualisation in Concept Design,” Des. Stud., 22(1), pp. 5–26. [CrossRef]
Kim, M. H., Kim, Y. S., Lee, H. S., and Park, J. A., 2007, “An Underlying Cognitive Aspect of Design Creativity: Limited Commitment Mode Control Strategy,” Des. Stud., 28(6), pp. 585–604. [CrossRef]
Takai, S., 2011, “A Pre-Post Assessment of Creativity Methods in an Experimental Course,” Int. J. Eng. Educ., 27(2), pp. 380–388.
White, A., and Smith, B. L., 2001, “Assessing Advertising Creativity Using the Creative Product Semantic Scale,” J. Advertising Res., 41(6), pp. 27–34.
White, A., Shen, F., and Smith, B. L., 2002, “Judging Advertising Creativity Using the Creative Product Semantic Scale,” J. Creative Behav., 36(4), pp. 241–253. [CrossRef]
Smith, B. L., 1993, “Interpersonal Behaviors that Damage the Productivity of Creative Problem Solving Groups,” J. Creative Behav., 27(3), pp. 171–187. [CrossRef]
Bytheway, C. W., 2007, FAST Creativity & Innovation: Rapidly Improving Processes, Product Development and Solving Complex Problems, J. Ross Pub., Fort Lauderdale, FL.
Olivero, M., 1990, “Get Crazy! How to Have a Breakthrough Idea,” Working Woman, 213(1), pp. 145–147, 222.
Osborn, A. F., 1963, Applied Imagination: Principles and Procedures of Creative Problem-Solving, Scribner, NY.
Pahl, G., and Beitz, W., 1996, Engineering Design—A Systematic Approach, Springer Limited, London, UK.
Ojala, M., and Garriga, G. C., 2010, “Permutation Tests for Studying Classifier Performance,” J. Mach. Learn. Res., 11(6), pp. 1833–1863.
Shrout, P. E., and Fleiss, J. L., 1979, “Intraclass Correlations: Uses in Assessing Rater Reliability,” Psychol. Bull., 86(2), pp. 420–428. [CrossRef] [PubMed]
Tseng, I., Cagan, J., Kotovsky, K., and Wood, W., 2012, “Form Function Fidelity,” ASME J. Mech. Des., 135(1), p. 011006. [CrossRef]
Hallgren, K. A., 2012, “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial,” Tutorials Quant. Methods Psychol., 8(1), pp. 23–34.
Cicchetti, D. V., 1994, “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology,” Psychol. Assess., 6(4), pp. 284–290. [CrossRef]
Multon, K., 2010, “Interrater Reliability,” Encyclopedia of Research Design, N.Salkind, ed., SAGE Publications, Inc., Thousand Oaks, CA, pp. 627–629.
Takai, S., and Ishii, K., 2010, “A Use of Subjective Clustering to Support Affinity Diagram Results in Customer Needs Analysis,” Concurrent Eng.: Res. Appl., 18(2), pp. 101–109. [CrossRef]


Grahic Jump Location
Fig. 1

Data acquisition and analysis procedure

Grahic Jump Location
Fig. 4

Project testing setup

Grahic Jump Location
Fig. 5

Project deliverables: (a) concept sketch, (b) POC prototype, and (c) final product

Grahic Jump Location
Fig. 6

Design concepts and testing results

Grahic Jump Location
Fig. 7

Correlations of performance scores and CPSS: (a) CPSS dimensions, (b) CPSS subscales, and (c) CPSS items

Grahic Jump Location
Fig. 8

Examples of correlations, regression lines, and p-values: (a) final products, (b) POC prototypes, and (c) concept sketches

Grahic Jump Location
Fig. 9

Correlations of CPSS: (a) CPSS dimensions, (b) CPSS subscales, and (c) CPSS items

Grahic Jump Location
Fig. 10

CPSS scores of concepts: (a) CPSS dimensions, (b) CPSS subscales, and (c) CPSS items

Grahic Jump Location
Fig. 11

Classification trees: (a) CPSS dimensions, (b) CPSS subscales, and (c) CPSS items

Grahic Jump Location
Fig. 12

Effectiveness of decision rules: (a) number of evaluation, (b) misclassification rate, (c) LOOCV error, and (d) permutation-based

Grahic Jump Location
Fig. 13

Creativity product semantic scale

Grahic Jump Location
Fig. 14

Performances of final products and the number of cognitive modes

Grahic Jump Location
Fig. 15

Intraclass Correlations of CPSS evaluations



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In