Research Papers: Design Theory and Methodology

Assessing Quality of User-Submitted Need Statements From Large-Scale Needfinding: Effects of Expertise and Group Size

[+] Author and Article Information
Cory R. Schaffhausen

Department of Mechanical Engineering,
University of Minnesota,
Minneapolis, MN 55455
e-mail: schaf390@umn.edu

Timothy M. Kowalewski

Department of Mechanical Engineering,
University of Minnesota,
Minneapolis, MN 55455
e-mail: timk@umn.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received March 30, 2015; final manuscript received September 9, 2015; published online October 15, 2015. Assoc. Editor: Carolyn Seepersad.

J. Mech. Des 137(12), 121102 (Oct 15, 2015) (9 pages) Paper No: MD-15-1262; doi: 10.1115/1.4031655 History: Received March 30, 2015; Revised September 09, 2015

Collecting data on user needs often results in a surfeit of candidate need statements. Additional analysis is necessary to prioritize a small subset for further consideration. Previous analytic methods have been used for small quantities (often fewer than 75 statements). This study presents a simplified quality metric and online interface appropriate to initially screen and prioritize lists exceeding 500 statements for a single topic or product area. Over 20,000 ratings for 1697 need statements across three common product areas were collected in 6 days. A series of hypotheses were tested: (1) Increasing the quantity of participants submitting needs increases the number of high-quality needs as judged by users; (2) increasing the quantity of needs contributed per person increases the number of high-quality needs as judged by users; and (3) increasing levels of self-rated user expertise will not significantly increase the number of high-quality needs per person. The results provided important quantitative evidence of fundamental relationships between the quantity and quality of need statements. Higher quantities of total needs submitted correlated to higher quantities of high-quality need statements both due to increasing group size and due to increasing counts per person using novel content-rich methods to help users articulate needs. Based on a multivariate analysis, a user's topic-specific expertise (self-rated) and experience level (self-rated hours per week) were not significantly associated with increasing quantities of high-quality needs.

Copyright © 2015 by ASME
Your Session has timed out. Please sign back in to continue.


Griffin, A. , and Hauser, J. R. , 1993, “ The Voice of the Customer,” Mark. Sci., 12(1), pp. 1–27. [CrossRef]
Ulwick, A. W. , 2002, “ Turn Customer Input Into Innovation,” Harv. Bus. Rev., 80(1), pp. 91–97. [PubMed]
Patnaik, D. , 2014, Needfinding: Design Research and Planning, 3rd ed., CreateSpace Independent Publishing Platform, Lexington, KY.
Lin, J. , and Seepersad, C. C. , 2007, “ Empathic Lead Users: The Effects of Extraordinary User Experiences on Customer Needs Analysis and Product Redesign,” ASME Paper No. DETC2007/35302.
Schaffhausen, C. R. , and Kowalewski, T. M. , 2015, “ Large-Scale Needfinding: Methods of Increasing User-Generated Needs From Large Populations,” ASME J. Mech. Des., 137(7), p. 071403. [CrossRef]
Vredenburg, K. , Mao, J. , Smith, P. , and Carey, T. , 2002, “ A Survey of User-Centered Design Practice,” SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves, pp. 471–478.
Von Hippel, E. , 1986, “ Lead Users: A Source of Novel Product Concepts,” Manage. Sci., 32(7), pp. 791–805. [CrossRef]
Bayus, B. L. , 2008, “ Understanding Customer Needs,” The Handbook of Technology and Innovation Management, S. Shane , ed., Wiley, West Sussex, UK, pp. 115–141.
Patnaik, D. , and Becker, R. , 1999, “ Needfinding: The Why and How of Uncovering People's Needs,” Des. Manage. J. (Former Ser.), 10(2), pp. 37–43. [CrossRef]
Money, A. , Barnett, J. , Kuljis, J. , Craven, M. , Martin, J. , and Young, T. , 2011, “ The Role of the User Within the Medical Device Design and Development Process: Medical Device Manufacturers' Perspectives,” BMC Med. Inf. Decis. Making, 11(15), pp. 1–12.
Kano, N. , Seraku, N. , Takahashi, F. , and Tsuji, S. , 1984, “ Attractive Quality and Must-Be Quality,” J. Jpn. Soc. Qual. Control, 14(2), pp. 147–156.
Mikulic, J. , and Prebezac, D. , 2011, “ A Critical Review of Techniques for Classifying Quality Attributes in the Kano Model,” Managing Serv. Qual.: Int. J., 21(1), pp. 46–66. [CrossRef]
Ulrich, K. T. , and Eppinger, S. D. , 2004, Product Design and Development, 3rd ed., McGraw-Hill/Irwin, New York.
Takai, S. , and Ishii, K. , 2010, “ A Use of Subjective Clustering to Support Affinity Diagram Results in Customer Needs Analysis,” Concurrent Eng., 18(2), pp. 101–109. [CrossRef]
Simpson, T. W. , Bobuk, A. , Slingerland, L. A. , Brennan, S. , Logan, D. , and Reichard, K. , 2012, “ From User Requirements to Commonality Specifications: An Integrated Approach to Product Family Design,” Res. Eng. Des., 23(2), pp. 141–153. [CrossRef]
Cormier, P. , Olewnik, A. , and Lewis, K. , 2014, “ Toward a Formalization of Affordance Modeling for Engineering Design,” Res. Eng. Des., 25(3), pp. 259–277. [CrossRef]
Ciavola, B. T. , Wu, C. , and Gershenson, J. K. , 2015, “ Integrating Function- and Affordance-Based Design Representations,” ASME J. Mech. Des., 137(5), p. 051101. [CrossRef]
Green, M. G. , Rajan, P. K. P. , and Wood, K. L. , 2004, “ Product Usage Context: Improving Customer Needs Gathering and Design Target Setting,” ASME Paper No. DETC2004-57498.
Scaravetti, D. , Nadeau, J.-P. , Pailhès, J. , and Sebastian, P. , 2005, “ Structuring of Embodiment Design Problem Based on the Product Lifecycle,” Int. J. Prod. Dev., 2(1–2), pp. 47–70. [CrossRef]
Ulwick, A. W. , 2005, What Customers Want: Using Outcome-Driven Innovation to Create Breakthrough Products and Services, Vol. 71408673, McGraw-Hill, New York.
Matzler, K. , and Hinterhuber, H. H. , 1998, “ How to Make Product Development Projects More Successful by Integrating Kano's Model of Customer Satisfaction Into Quality Function Deployment,” Technovation, 18(1), pp. 25–38. [CrossRef]
Ma, X. , Yu, L. , Forlizzi, J. L. , and Dow, S. P. , 2015, “ Exiting the Design Studio: Leveraging Online Participants for Early-Stage Design Feedback,” 18th ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 676–685.
Dean, D. L. , Hender, J. M. , Rodgers, T. L. , and Santanen, E. L. , 2006, “ Identifying Quality, Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation,” J. Assoc. Inf. Syst., 7(10), pp. 646–698.
Chen, C.-C. , and Chuang, M.-C. , 2008, “ Integrating the Kano Model Into a Robust Design Approach to Enhance Customer Satisfaction With Product Design,” Int. J. Prod. Econ., 114(2), pp. 667–681. [CrossRef]
Sharif Ullah, A. M. M. , and Tamaki, J. , 2011, “ Analysis of Kano-Model-Based Customer Needs for Product Development,” Syst. Eng., 14(2), pp. 154–172. [CrossRef]
Zahedi, F. , 1986, “ The Analytic Hierarchy Process: A Survey of the Method and Its Applications,” Interfaces, 16(4), pp. 96–108. [CrossRef]
Duhovnik, J. , Kušar, J. , Starbek, M. , and Tomaževič, R. , 2006, “ Development Process With Regard to Customer Requirements,” Concurrent Eng., 14(1), pp. 67–82. [CrossRef]
Agouridas, V. , Winand, H. , McKay, A. , and de Pennington, A. , 2006, “ Early Alignment of Design Requirements With Stakeholder Needs,” Proc. Inst. Mech. Eng., Part B, 220(9), pp. 1483–1507. [CrossRef]
Paolacci, G. , Chandler, J. , and Ipeirotis, P. , 2010, “ Running Experiments on Amazon Mechanical Turk,” Judgment Decis. Making, 5(5), pp. 411–419.
Peer, E. , Vosgerau, J. , and Acquisti, A. , 2014, “ Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk,” Behav. Res. Methods, 46(4), pp. 1023–1031. [CrossRef] [PubMed]
Schaffhausen, C. R. , and Kowalewski, T. M. , 2015, “ Large Scale Needs-Based Open Innovation Via Automated Semantic Textual Similarity Analysis,” ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (in press).
Agirre, E. , Diab, M. , Cer, D. , and Gonzalez-Agirre, A. , 2012, “ Semeval-2012 Task 6: A Pilot on Semantic Textual Similarity,” Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, Montréal, Canada, Association of Computational Linguistics, Stroudsburg, PA, pp. 385–393.
Bretz, F. , Hothorn, T. , and Westfall, P. , 2010, Multiple Comparisons Using R, CRC, Boca Raton, FL.
Diehl, M. , and Stroebe, W. , 1987, “ Productivity Loss in Brainstorming Groups: Toward the Solution of a Riddle,” J. Pers. Soc. Psychol., 53(3), pp. 497–509. [CrossRef]
Dugosh, K. L. , and Paulus, P. B. , 2005, “ Cognitive and Social Comparison Processes in Brainstorming,” J. Exp. Soc. Psychol., 41(3), pp. 313–320. [CrossRef]
Paulus, P. B. , Kohn, N. W. , and Arditti, L. E. , 2011, “ Effects of Quantity and Quality Instructions on Brainstorming,” J. Creat. Behav., 45(1), pp. 38–46. [CrossRef]
Kudrowitz, B. M. , and Wallace, D. , 2013, “ Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation,” J. Eng. Des., 24(2), pp. 120–139. [CrossRef]
Vernon, D. , and Hocking, I. , 2014, “ Thinking Hats and Good Men: Structured Techniques in a Problem Construction Task,” Thinking Skills Creat., 14, pp. 41–46. [CrossRef]


Grahic Jump Location
Fig. 2

Process for analysis of one group size permutation

Grahic Jump Location
Fig. 1

Overview of previous (unshaded) and current (shaded) data collection and analysis

Grahic Jump Location
Fig. 3

Stacked distribution of quality scores (all phases)

Grahic Jump Location
Fig. 9

Mean ratings for differences in submitter and rater experience (negative difference: need from low-experience user-rated by high-experience user), group sizes: *for n < 20; for n > 100

Grahic Jump Location
Fig. 4

High-quality needs for increasing group sizes (error bars indicate standard errors)

Grahic Jump Location
Fig. 5

High-quality needs (cutoff score = 7.5) for all topics and group sizes (error bars indicate standard errors)

Grahic Jump Location
Fig. 6

Top quartile needs for users with increasing total need counts (shading indicates 95% CI)

Grahic Jump Location
Fig. 7

Top quartile needs for all topics and expertise groups (group size, n, shown)

Grahic Jump Location
Fig. 8

Top quartile needs for all topics and experience groups (group size, n, shown)




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In