Research Papers: Design Theory and Methodology

Assessing Quality of User-Submitted Need Statements From Large-Scale Needfinding: Effects of Expertise and Group Size

[+] Author and Article Information
Cory R. Schaffhausen

Department of Mechanical Engineering,
University of Minnesota,
Minneapolis, MN 55455
e-mail: schaf390@umn.edu

Timothy M. Kowalewski

Department of Mechanical Engineering,
University of Minnesota,
Minneapolis, MN 55455
e-mail: timk@umn.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received March 30, 2015; final manuscript received September 9, 2015; published online October 15, 2015. Assoc. Editor: Carolyn Seepersad.

J. Mech. Des 137(12), 121102 (Oct 15, 2015) (9 pages) Paper No: MD-15-1262; doi: 10.1115/1.4031655 History: Received March 30, 2015; Revised September 09, 2015

Collecting data on user needs often results in a surfeit of candidate need statements. Additional analysis is necessary to prioritize a small subset for further consideration. Previous analytic methods have been used for small quantities (often fewer than 75 statements). This study presents a simplified quality metric and online interface appropriate to initially screen and prioritize lists exceeding 500 statements for a single topic or product area. Over 20,000 ratings for 1697 need statements across three common product areas were collected in 6 days. A series of hypotheses were tested: (1) Increasing the quantity of participants submitting needs increases the number of high-quality needs as judged by users; (2) increasing the quantity of needs contributed per person increases the number of high-quality needs as judged by users; and (3) increasing levels of self-rated user expertise will not significantly increase the number of high-quality needs per person. The results provided important quantitative evidence of fundamental relationships between the quantity and quality of need statements. Higher quantities of total needs submitted correlated to higher quantities of high-quality need statements both due to increasing group size and due to increasing counts per person using novel content-rich methods to help users articulate needs. Based on a multivariate analysis, a user's topic-specific expertise (self-rated) and experience level (self-rated hours per week) were not significantly associated with increasing quantities of high-quality needs.

Copyright © 2015 by ASME
Your Session has timed out. Please sign back in to continue.



Grahic Jump Location
Fig. 1

Overview of previous (unshaded) and current (shaded) data collection and analysis

Grahic Jump Location
Fig. 2

Process for analysis of one group size permutation

Grahic Jump Location
Fig. 3

Stacked distribution of quality scores (all phases)

Grahic Jump Location
Fig. 9

Mean ratings for differences in submitter and rater experience (negative difference: need from low-experience user-rated by high-experience user), group sizes: *for n < 20; for n > 100

Grahic Jump Location
Fig. 8

Top quartile needs for all topics and experience groups (group size, n, shown)

Grahic Jump Location
Fig. 7

Top quartile needs for all topics and expertise groups (group size, n, shown)

Grahic Jump Location
Fig. 6

Top quartile needs for users with increasing total need counts (shading indicates 95% CI)

Grahic Jump Location
Fig. 5

High-quality needs (cutoff score = 7.5) for all topics and group sizes (error bars indicate standard errors)

Grahic Jump Location
Fig. 4

High-quality needs for increasing group sizes (error bars indicate standard errors)




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In