Research Papers: Design Theory and Methodology

But Is It Creative? Delineating the Impact of Expertise and Concept Ratings on Creative Concept Selection

[+] Author and Article Information
Christopher A. Gosnell

The Pennsylvania State University,
112 Leonhard Building,
University Park, PA 16802
e-mail: cag5266@psu.edu

Scarlett R. Miller

The Pennsylvania State University,
213-P Hammond Building,
University Park, PA 16802-1401
e-mail: scarlettmiller@psu.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received January 7, 2015; final manuscript received September 24, 2015; published online December 4, 2015. Assoc. Editor: Kristina Shea.

J. Mech. Des 138(2), 021101 (Dec 04, 2015) (11 pages) Paper No: MD-15-1015; doi: 10.1115/1.4031904 History: Received January 07, 2015; Revised September 24, 2015

While creativity is often stressed in the conceptual phases of design, it is rarely considered during the concept selection process. Before effective methods can be developed to aid in creative concept section, however, differences in the perceptions of creativity between expert and novice designers and the influence of creativity evaluation methods on the process must be considered. Therefore, this paper was developed to address these questions by studying 11 expert and 11 novice designers. Specifically the study was developed to understand if experts' and novices' perception of a concepts creativity aligned, to introduce and compare the utility of our tool for assessing semantic creativity (TASC) to existing creativity evaluation methods, and to identify if our TASC method could be used as a proxy for expert evaluators. Our findings reveal that experts and novices generally had similar perceptions of a concept's creativity and that the TASC method was tapping into similar constructs of human perceptions of concept creativity. The results of this study contribute to our understanding of the factors that influence the selection or filtering of creative ideas after idea generation and provide a framework for research in this field.

Copyright © 2016 by ASME
Topics: Creativity , Design
Your Session has timed out. Please sign back in to continue.


Sarkar, P. , and Chakrabarti, A. , 2011, “ Assessing Design Creativity,” Des. Stud., 32(4), pp. 348–383. [CrossRef]
Chulvi, V. , Gonzalez-Cruz, M. C. , Mulet, E. , and Aguilar-Zambrano, J. , 2012, “ Influence of Type of Idea-Generation Method on the Creativity of Solutions,” Res. Eng. Des., 24(1), pp. 33–41. [CrossRef]
Yilmaz, S. , Seifert, C. M. , and Gonzalez, R. , 2012, “ Design Heuristics: Cognitive Strategies for Creativity in Idea Generation,” Design Computing and Cognition '10, J. S. Gero, ed., Springer, New York, pp. 35–53.
Shah, J. J. , Vargas-Hernandez, N. , and Smith, S. M. , 2003, “ Metrics for Measuring Ideation Effectiveness,” Des. Stud., 24(1), pp. 111–134. [CrossRef]
Rietzchel, E. F. , Nijstad, B. A. , and Stroebe, W. , 2006, “ Productivity is Not Enough: A Comparison of Interactive and Nominal Groups in Idea Generation and Selection,” J. Exp. Soc. Psychol., 42(2), pp. 244–251. [CrossRef]
Snider, C. , Cash, P. , Dekoninck, E. , and Culley, S. , 2012, “ Variation in Creative Behaviour During the Later Stages of the Design Process,” ICDC2012: The 2nd International Conference on Design Creativity, University of Bath, pp. 147–156.
Kudrowitz, B. M. , and Wallace, D. , 2013, “ Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation,” J. Eng. Des., 24(2), pp. 120–139. [CrossRef]
Fishburn, P. C. , 1970, “ Utility Theory for Decision Making,” DTIC Document, Mclean, VA, Report No. RAC-R-105.
Von Neumann, J. , and Morgenstern, O. , 1953, Theory of Games and Economic Behavior, 3rd ed., Princeton University Press, Princeton, NJ.
Saaty, T. L. , 1988, What is the Analytic Hierarchy Process?, Springer, Berlin, Germany.
Saaty, T. L. , 1980, The Analytic Hierarchy Process, McGraw Hill, New York.
Pugh, S. , 1991, Total Design: Integrated Methods for Successful Product Engineering, Addison-Wesley, Workingham, UK.
Pugh, S. , 1981, “ Concept Selection—A Method That Works,” International Conference of Engineering Design, Rome, pp. 497–506.
Ter Harr, S. , Clausling, D. , and Eppinger, S. , 1993, “ Integration of Quality Function Deployment in the Design Structure Matrix,” MIT, Cambridge, MA, Working Paper.
Hauser, J. , and Clausing, D. , 1996, “ The House of Quality,” IEEE Eng. Manage. Rev., 24(1), pp. 24–32.
Ross, T. , 1995, Fuzzy Logic With Engineering Applications, McGraw Hill, West Sussex, U.K.
Thurston, D. L. , and Carnahan, J. V. , 1992, “ Fuzzy Ratings and Utility Analysis in Preliminary Design Evaluation of Multiple Attributes,” ASME J. Mech. Des., 114(4), pp. 648–658. [CrossRef]
Okugan, G. E. , and Tauhid, S. , 2008, “ Concept Selection Methods—A Literature Review From 1980 to 2008,” Int. J. Des. Eng., 1(3), pp. 243–277.
Oman, S. K. , Tumer, I . Y. , Wood, K. , and Seepersad, C. , 2013, “ A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects,” Res. Eng. Des., 24(1), pp. 65–92. [CrossRef]
Amabile, T. M. , 1983, “ The Social Psychology of Creativity: A Componential Conceptualization,” J. Pers. Soc. Psychol., 45(2), pp. 357–376. [CrossRef]
Brown, D. , 2013, “ Developing Computational Design Creativity Systems,” Int. J. Des. Creativity Innovation, 1(1), pp. 43–55. [CrossRef]
Besemer, S. P. , and O'Quin, K. , 1999, “ Confirming the Three-Factor Creative Product Analysis Matrix Model in an American Sample,” Creativity Res. J., 12(4), pp. 287–296. [CrossRef]
Kaufman, J. C. , Baer, J. , Cole, J. C. , and Sexton, J. D. , 2008, “ A Comparison of Expert and Nonexpert Raters Using the Consensual Assessment Technique,” Creativity Res. J., 20(2), pp. 171–178. [CrossRef]
Fischer, G. , 2013, “ Learning, Social Creativity, and Cultures of Participation,” Learning and Collective Creativity: Activity–Theoretical and Sociocultural Studies, Routledge, New York, p. 198.
Martin, M. W. , 2006, “ Moral Creativity in Science and Engineering,” Sci. Eng. Ethics, 12(3), pp. 421–433. [CrossRef] [PubMed]
Nelson, B. A. , Wilson, J. O. , Rosen, D. , and Yen, J. , 2009, “ Refined Metrics for Measuring Ideation Effectiveness,” Des. Stud., 30(6), pp. 737–743. [CrossRef]
Chulvi, V. , Mulet, E. , Chakrabarti, A. , López-Mesa, B. , and González-Cruz, C. , 2012, “ Comparison of the Degree of Creativity in the Design Outcomes Using Different Design Methods,” J. Eng. Des., 23(4), pp. 241–269. [CrossRef]
Srivathsavai, R. , Genco, N. , Holtta-Otto, K. , and Seepersad, C. , 2010, “ Study of Existing Metrics Used in Measurement of Ideation Effectiveness,” ASME Paper No. DETC2010-28802.
Redmond, M. R. , Mumford, M. D. , and Teach, R. , 1993, “ Putting Creativity to Work: Effects of Leader Behavior on Subordinate Creativity,” Organ. Behav. Hum. Decis. Processes, 55(1), pp. 120–151. [CrossRef]
Bedell-Avers, K. , Hunter, S. T. , Angie, A. D. , Eubanks, D. L. , and Mumford, M. D. , 2009, “ Charismatic, Ideological, and Pragmatic Leaders: An Examination of Leader–Leader Interactions,” Leadership Q., 20(3), pp. 299–315. [CrossRef]
Hunter, S. T. , Bedell, K. E. , Ligon, G. S. , Hunsicker, C. M. , and Mumford, M. D. , 2008, “ Applying Multiple Knowledge Structures in Creative Thought: Effects on Idea Generation and Problem-Solving,” Creativity Res. J., 20(2), pp. 137–154. [CrossRef]
Brown, D. C. , 2015, “ Computational Design Creativity Evaluation,” Design Computing and Cognition '14, J. S. Gero and S. Hanna, eds., Springer International Publishing, New York, pp. 207–224.
Maher, M. L. , and Fisher, D. H. , 2012, “ Using AI to Evaluate Creative Designs,” 2nd International Conference on Design Creativity, Glasgow, UK, pp. 45–54.
Gero, J. S. , and Kannengiesser, U. , 2007, “ Locating Creativity in a Framework of Designing for Innovation,” Trends in Computer Aided Innovation, Springer, New York, pp. 57–66.
Shiffrin, R. M. , and Schneider, W. , 1977, “ Controlled and Automatic Human Information Processing: II. Perceptual Learning, Automatic Attending and a General Theory,” Psychol. Rev., 84(2), pp.127–190. [CrossRef]
Baddeley, A. D. , 2002, “ Is Working Memory Still Working?,” Eur. Psychol., 7(2), pp. 85–97. [CrossRef]
Licuanan, B. F. , Dailey, L. R. , and Mumford, M. D. , 2007, “ Idea Evaluation: Error in Evaluating Highly Original Ideas,” J. Creative Behav., 41(1), pp. 1–27. [CrossRef]
Green, M. , Seepersad, C. , and Hölttä-Otto, K. , 2014, “ Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study,” ASME Paper No. DETC2014-34434.
Cross, N. , 2004, “ Expertise in Design: An Overview,” Des. Stud., 25(5), pp. 427–441. [CrossRef]
Atman, C. J. , Adams, R. S. , Cardella, M. E. , Turns, J. , Mosborg, S. , and Saleem, J. , 2007, “ Engineering Design Processes: A Comparison of Students and Expert Practitioners,” J. Eng. Educ., 96(4), pp. 359–379. [CrossRef]
Worsley, M. , and Blikstein, P. , 2011, “ What's an Expert? Using Learning Analytics to Identify Emergent Markers of Expertise Through Automated Speech, Sentiment and Sketch Analysis,” EDM, pp. 235–240.
Ericsson, K. A. , and Lehmann, A. C. , 1996, “ Expert and Exceptional Performance: Evidence of Maximal Adaptation to Task Constraints,” Annu. Rev. Psychol., 47(1), pp. 273–305. [CrossRef] [PubMed]
Jiao, J. R. , Zhang, Y. , and Helander, M. , 2006, “ A Kansei Mining System for Affective Design,” Expert Syst. Appl., 30(4), pp. 658–673. [CrossRef]
Han, S. H. , and Hong, S. W. , 2003, “ A Systematic Approach for Coupling User Satisfaction With Product Design,” Ergonomics, 46(13–14), pp. 1441–1461. [CrossRef] [PubMed]
Chuang, M.-C. , and Ma, Y.-C. , 2001, “ Expressing the Expected Product Images in Product Design of Micro-Electronic Products,” Int. J. Ind. Ergon., 27(4), pp. 233–245. [CrossRef]
Benedek, J. , and Miner, T. , 2002, “ Measuring Desirability: New methods for Evaluating Desirability in a Usability Lab Setting,” Usability Professionals Association, 2003, pp. 8–12.
Korpershoek, H. , Kuyper, H. , Werf, G. v. d. , and Bosker, R. , 2010, “ Who ‘Fits’ the Science and Technology Profile? Personality Differences in Secondary Education,” J. Res. Pers., 44(5), pp. 649–654. [CrossRef]
Choi, K. , and Jun, C. , 2007, “ A Systematic Approach to the Kansei Factors of Tactile Sense Regarding the Surface Roughness,” Appl. Ergon., 38(1), pp. 53–63. [CrossRef] [PubMed]
Childs, T. , Agouridas, V. , Barnes, C. , and Henson, B. , 2006, “ Controlled Appeal Product Design: A Life Cycle Role for Affective (Kansei) Engineering,” LCE2006, pp. 537–542.
Prabhakaran, R. , Green, A. E. , and Gray, J. R. , 2013, “ Thin Slices of Creativity: Using Single-Word Utterances to Assess Creative Cognition,” Behavior Research Methods, Springer, New York, pp. 1–19.
Fabiani, M. , and Donchin, E. , 1995, “ Encoding Processes and Memory Organization: A Model of the Von Restorff Effect,” J. Exp. Psychol., 21(1), pp. 224–240.
Snyder, K. A. , Blank, M. P. , and Marsolek, C. J. , 2008, “ What Form of Memory Underlies Novelty Preferences?,” Psychon. Bull. Rev., 15(2), pp. 315–321. [CrossRef] [PubMed]
Anderson, C. J. , Glassman, M. , McAfee, R. B. , and Pinelli, T. , 2001, “ An Investigation of Factors Affecting How Engineers and Scientists Seek Information,” J. Eng. Technol. Manage., 18(2), pp. 131–155. [CrossRef]
Peracchio, L. A. , and Tybout, A. M. , 1996, “ The Moderating Role of Prior Knowledge in Schema-Based Product Evaluation,” J. Consum. Res., 23(3), pp. 177–192. [CrossRef]
Kudrowitz, B. , and Dippo, C. , 2013, “ Getting to the Novel Ideas: Exploring the Alternative Uses Test of Divergent Thinking,” ASME Paper No. DETC2013-13262.
Nikander, J. B. , Liikkanen, L. A. , and Laakso, M. , 2014, “ The Preference Effect in Design Concept Evaluation,” Des. Stud., 35(5), pp. 473–499. [CrossRef]
Lu, C.-C. , and Luh, D.-B. , 2012, “ A Comparison of Assessment Methods and Raters in Product Creativity,” Creativity Res. J., 24(4), pp. 331–337. [CrossRef]
Von Hippel, E. , 1986, “ Lead Users: A Source of Novel Product Concepts,” Manage. Sci., 32(7), pp. 791–805. [CrossRef]
Dailey, L. , and Mumford, M. D. , 2006, “ Evaluative Aspects of Creative Thought: Errors in Appraising the Implications of New Ideas,” Creativity Res. J., 18(3), pp. 385–390. [CrossRef]
Toh, C. , and Miller, S. R. , 2013, “ Product Dissection or Visual Inspection? The Impact of Designer–Product Interactions on Engineering Design Creativity,” ASME Design Engineering Technical Conferences, Portland, OR, p. V005T06A011.
Toh, C. , Miller, S. , and Okudan Kremer, G. , 2012, “ Mitigating Design Fixation Effects in Engineering Design Through Product Dissection Activities,” Design Computing and Cognition '12, J. S. Gero, ed., Springer, New York, pp. 95–113.
Miller, S. R. , Bailey, B. P. , and Kirlik, A. , 2014, “ Exploring the Utility of Bayesian Truth Serum for Assessing Design Knowledge,” Hum. Comput. Interact., 29(5–6), pp. 487–515. [CrossRef]
Williams, D. , Kelly, G. , and Anderson, L. , “ MSN 9: New User-Centered Desirability Methods Produce Compelling Visual Design,” CHI'04 Extended Abstracts on Human Factors in Computing Systems, ACM, New York, pp. 959–974.
Kolb, P. , “ Experiments on the Difference Between Semantic Similarity and Relatedness,” 17th Nordic Conference on Computational Linguistics-NODALIDA’09, pp. 1–8.
Kolb, P. , 2008, “ DISCO: A Multilingual Database of Distributionally Similar Words,” KONVENS-2008, Berlin, pp. 1–12.
Linsey, J. S. , Clauss, E. F. , Kurtoglu, T. , Murphy, J. T. , Wood, K. L. , and Markman, A. B. , 2011, “ An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods,” ASME J. Mech. Des., 133(3), p. 031008. [CrossRef]
Wells, J. D. , Campbell, D. E. , Valacich, J. S. , and Featherman, M. , 2010, “ The Effect of Perceived Novelty on the Adoption of Information Technology Innovations: A Risk/Reward Perspective,” Decis. Sci., 41(4), pp. 813–843. [CrossRef]
Endicott, J. , Spitzer, R. L. , Fleiss, J. L. , and Cohen, J. , 1976, “ The Global Assessment Scale: A Procedure for Measuring Overall Severity of Psychiatric Disturbance,” Arch. Gen. Psychiatry, 33(6), pp. 766–771. [CrossRef] [PubMed]
Chiou, C.-F. , Hay, J. W. , Wallace, J. F. , Bloom, B. S. , Neumann, P. J. , Sullivan, S. D. , Yu, H.-T. , Keeler, E. B. , Henning, J. M. , and Ofman, J. J. , 2003, “ Development and Validation of a Grading System for the Quality of Cost-Effectiveness Studies,” Med. Care, 41(1), pp. 32–44. [CrossRef] [PubMed]
Joyce, M. , and Kirakowski, J. , 2014, “ Measuring Confidence in Internet Use: The Development of an Internet Self-efficacy Scale,” Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience, Springer, New York, pp. 250–260.
Landis, J. R. , and Koch, G. G. , 1977, “ The Measurement of Observer Agreement for Categorical Data,” Biometrics, 33(1), pp. 159–174. [CrossRef] [PubMed]
Chan, S. , 1982, “ Expert Judgments Made Under Uncertainty: Some Evidence and Suggestions,” Soc. Sci. Q., 63, pp. 428–444.
Anderson, J. R. , 1987, “ Skill Acquisition: Compilation of Weak-Method Problem Situations,” Psychol. Rev., 94(2), pp. 192–210. [CrossRef]
Bollegala, D. , Matsuo, Y. , and Ishizuka, M. , 2011, “ A Web Search Engine-Based Approach to Measure Semantic Similarity Between Words,” IEEE Trans. Knowl. Data Eng., 23(7), pp. 977–990. [CrossRef]
Wu, W. , Luther, K. , Pavel, A. , Hartmann, B. , Dow, S. , and Agrawala, M. , 2013, CrowdCritter: Strategies for Crowdsourcing Visual Design Critique.
Burnap, A. , Ren, Y. , Papalambros, P. Y. , Gonzalez, R. , and Gerth, R. , 2013, “ A Simulation Based Estimation of Crowd Ability and Its Influence on Crowdsourced Evaluation of Design Concepts,” ASME Paper No. DETC2013-13020.
Rietzschel, E. , Nijstad, B. A. , and Stroebe, W. , 2010, “ The Selection of Creative Ideas After Individual Idea Generation: Choosing Between Creativity and Impact,” Br. J. Psychol., 101(1), pp. 47–68. [CrossRef] [PubMed]
Mueller, J. S. , Melwani, S. , and Goncalo, J. A. , 2011, “ The Bias Against Creativity: Why People Desire but Reject Creative Ideas,” Psychol. Sci., 2011, p. 0956797611421018.


Grahic Jump Location
Fig. 1

Summary comparisons of design evaluations from “toothbrush” design problem

Grahic Jump Location
Fig. 2

The inter-rater reliability (Kappa) for the idea rating methods used in the current study based on the 22 raters

Grahic Jump Location
Fig. 3

A summary of the Cohen's weighted Kappa between novice rater's perception, novice TASC and SVS scores of all 27 designs

Grahic Jump Location
Fig. 4

A summary of the Cohen's weighted Kappa analysis between expert rater's perception, expert TASC and SVS scores of all 27 designs. ** Significant at p < 0.01, *p < 0.05.

Grahic Jump Location
Fig. 5

A summary of the Cohen's weighted Kappa calculations between expert perception and both novice TASC and novice perception scores by design problem



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In