Research Papers: Design Automation

Analyzing Participant Behaviors in Design Crowdsourcing Contests Using Causal Inference on Field Data

[+] Author and Article Information
Ashish M. Chaudhari, Jitesh H. Panchal

School of Mechanical Engineering,
Purdue University,
West Lafayette, IN 47907

Zhenghui Sha

Department of Mechanical Engineering,
University of Arkansas,
Fayetteville, AR 72701

Contributed by the Design Automation Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received July 5, 2017; final manuscript received April 25, 2018; published online June 8, 2018. Assoc. Editor: Harrison M. Kim.

J. Mech. Des 140(9), 091401 (Jun 08, 2018) (12 pages) Paper No: MD-17-1452; doi: 10.1115/1.4040166 History: Received July 05, 2017; Revised April 25, 2018

Crowdsourcing is the practice of getting ideas and solving problems using a large number of people on the Internet. It is gaining popularity for activities in the engineering design process ranging from concept generation to design evaluation. The outcomes of crowdsourcing contests depend on the decisions and actions of participants, which in turn depend on the nature of the problem and the contest. For effective use of crowdsourcing within engineering design, it is necessary to understand how the outcomes of crowdsourcing contests are affected by sponsor-related, contest-related, problem-related, and individual-related factors. To address this need, we employ existing game-theoretic models, empirical studies, and field data in a synergistic way using the theory of causal inference. The results suggest that participants' decisions to participate are negatively influenced by higher task complexity and lower reputation of sponsors. However, they are positively influenced by the number of prizes and higher allocation to prizes at higher levels. That is, an amount of money on any following prize generates higher participation than the same amount of money on the first prize. The contributions of the paper are: (a) a causal graph that encodes relationships among factors affecting crowdsourcing contests, derived from game-theoretic models and empirical studies, and (b) a quantification of the causal effects of these factors on the outcomes of GrabCAD, Cambridge, MA contests. The implications of these results on the design of future design crowdsourcing contests are discussed.

Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.


Chesbrough, H. W. , 2006, Open Innovation: The New Imperative for Creating and Profiting From Technology, Harvard Business Press, Boston, MA.
Panchal, J. H. , 2015, “ Using Crowds in Engineering Design Towards a Holistic Framework,” International Conference on Engineering Design, Design Society, Milan, Italy, July 27–30, pp. 27–30.
Che, Y.-K. , and Gale, I. , 2003, “ Optimal Design of Research Contests,” Am. Econ. Rev., 93(3), pp. 646–671. [CrossRef]
Taylor, C. , 1995, “ Digging for Golden Carrots: An Analysis of Research Tournaments,” Am. Econ. Rev., 85(4), pp. 872–890.
Burnap, A. , Ren, Y. , Gerth, R. , Papazoglou, G. , Gonzalez, R. , and Papalambros, P. Y. , 2015, “ When Crowdsourcing Fails: A Study of Expertise on Crowdsourced Design Evaluation,” ASME J. Mech. Des., 137(3), p. 031101. [CrossRef]
Green, M. , Seepersad, C. C. , and Hölttä-Otto, K. , 2014, “ Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study,” ASME Paper No. DETC2014-34434.
Gerth, R. J. , Burnap, A. , and Papalambros, P. , 2012, “ Crowdsourcing: A Primer and Its Implications for Systems Engineering,” Defense Technical Information Center, Warren, MI, Technical Report, No. 23063.
Kudrowitz, B. M. , and Wallace, D. , 2013, “ Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation,” J. Eng. Des., 24(2), pp. 120–139. [CrossRef]
Mturk, 2018, “ Human Intelligence Through an API,” Amazon Mechanical Turk Inc., Seattle, WA, accessed May 11, 2018, https://www.mturk.com
Corchón, L. C. , 2007, “ The Theory of Contests: A Survey,” Rev. Econ. Des., 11(2), pp. 69–100.
Szymanski, S. , and Valletti, T. M. , 2005, “ Incentive Effects of Second Prizes,” Eur. J. Political Econ., 21(2), pp. 467–481. [CrossRef]
Fullerton, R. , Lincster, B. , McKee, M. , and Slate, S. , 2002, “ Using Auctions to Reward Tournament Winners: Theory and Experimental Investigation,” RAND J. Econ., 33(1), pp. 62–84. [CrossRef]
Sheremata, R. M. , 2011, “ Contest Design: An Experimental Investigation,” Econ. Inquiry, 49(2), pp. 573–590. [CrossRef]
Sha, Z. , Kannan, K. N. , and Panchal, J. H. , 2015, “ Behavioral Experimentation and Game Theory in Engineering Systems Design,” ASME J. Mech. Des., 137(5), p. 051405. [CrossRef]
Li, K. , Xiao, J. , Wang, Y. , and Wang, Q. , 2013, “ Analysis of the Key Factors for Software Quality in Crowdsourcing Development: An Empirical Study on Topcoder.com,” IEEE 37th Annual Computer Software and Applications Conference (COMPSAC), Kyoto, Japan, July 22–26, pp. 812–817.
Kittur, A. , Chi, E. H. , and Suh, B. , 2008, “ Crowdsourcing User Studies With Mechanical Turk,” SIGCHI Conference on Human Factors in Computing Systems, (CHI) Florence, Italy, Apr. 5–10, pp. 453–456.
Boudreau, K. J. , Lacetera, N. , and Lakhani, K. R. , 2011, “ Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis,” Manage. Sci., 57(5), pp. 843–863. [CrossRef]
Archak, N. , 2010, “ Money, Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on Topcoder.com,” 19th International Conference on World Wide Web, Raleigh, NC, Apr. 26–30, pp. 21–30.
Pearl, J. , 2009, “ Causal Inference in Statistics: An Overview,” Stat. Surveys, 3, pp. 96–146. [CrossRef]
GrabCAD, 2018, “ Engineering & Design Challenges,” GrabCAD Inc., Cambridge, MA, accessed May 11, 2018, https://www.grabcad.com/challenges
Terwiesch, C. , and Xu, Y. , 2008, “ Innovation Contests, Open Innovation, and Multiagent Problem Solving,” Manage. Sci., 54(9), pp. 1529–1543. [CrossRef]
Clark, D. J. , and Riis, C. , 1998, “ Competition Over More Than One Prize,” Am. Econ. Rev., 88(1), pp. 276–289.
Loch, C. H. , Terwiesch, C. , and Thomke, S. , 2001, “ Parallel and Sequential Testing of Design Alternatives,” Manage. Sci., 47(5), pp. 663–678. [CrossRef]
Skaperdas, S. , 1996, “ Contest Success Functions,” Econ. Theory, 7(2), pp. 283–290. [CrossRef]
Schottner, A. , 2008, “ Fixed-Prize Tournaments Versus First-Price Auctions in Innovation Contests,” Econ. Theory, 35(1), pp. 55–71. [CrossRef]
Easley, D. , and Ghosh, A. , 2015, “ Behavioral Mechanism Design: Optimal Crowdsourcing Contracts and Prospect Theory,” 16th ACM Conference on Economics and Computation (EC), Portland, OR, June 15–19, pp. 679–696.
Sisak, D. , 2009, “ Multiple-Prize Contests—The Optimal Allocation of Prizes,” J. Econ. Surv., 23(1), pp. 82–114. [CrossRef]
Zheng, H. , Li, D. , and Hou, W. , 2011, “ Task Design, Motivation, and Participation in Crowdsourcing Contests,” Int. J. Electron. Commer., 15(4), pp. 57–88. [CrossRef]
Zheng, H. , Xie, Z. , Hou, W. , and Li, D. , 2014, “ Antecedents of Solution Quality in Crowdsourcing: The Sponsor's Perspective,” J. Electron. Commer. Res., 15(3), pp. 212–224.
Summers, J. D. , and Shah, J. J. , 2010, “ Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability,” ASME J. Mech. Des., 132(2), p. 021004. [CrossRef]
ElMaraghy, W. , ElMaraghy, H. , Tomiyama, T. , and Monostori, L. , 2012, “ Complexity in Engineering Design and Manufacturing,” CIRP Ann. Manuf. Technol., 61(2), pp. 793–814. [CrossRef]
Suh, N. P. , 1990, The Principles of Design (Oxford Series on Advanced Manufacturing), Oxford University Press, New York.
Kittur, A. , Smus, B. , Khamkar, S. , and Kraut, R. E. , 2011, “ Crowdforge: Crowdsourcing Complex Work,” 24th Annual ACM Symposium on User Interface Software and Technology (UIST), Santa Barbara, CA, Oct. 16–19, pp. 43–52.
Algesheimer, R. , Borle, S. , Dholakia, U. M. , and Singh, S. S. , 2010, “ The Impact of Customer Community Participation on Customer Behaviors: An Empirical Investigation,” Mark. Sci., 29(4), pp. 756–769. [CrossRef]
MacCallum, R. C. , 1995, Structural Equation Modeling: Concepts, Issues, and Applications (Model Specification: Procedures, Strategies, and Related Issues), Sage Publications, Thousand Oaks, CA, pp. 16–36.
Dorst, K. , 2004, “ On the Problem of Design Problems-Problem Solving and Design Expertise,” J. Des. Res., 4(2), pp. 185–196.
Scrapy, 2018, “A Fast and Powerful Scraping and Web Crawling Framework,” Scrapinghub, Ballincollig, Ireland, accessed May 11, 2018, https://www.scrapy.org
Alcoa, 2016, “Airplane Bearing Bracket Challenge,” GrabCAD.com, Cambridge, MA, accessed May 11, 2018, https://grabcad.com/challenges/airplane-bearing-bracket-challenge
Autodesk, 2013, “Autodesk Robot Gripper Arm Design Challenge,” GrabCAD.com, Cambridge, MA, accessed May 11, 2018, https://grabcad.com/challenges/autodesk-robot-gripper-arm-design-challenge
Rabaconda, 2012, “Dirtbike Tire Changing Tool Challenges,” GrabCAD.com, Cambridge, MA, accessed May 11, 2018, https://grabcad.com/challenges/dirtbike-tire-changing-tool-challenge
Alexa, 2018, “Alexa Website Traffic Statistics Tool,” Alexa Internet Inc., San Francisco, CA, accessed May 11, 2018, https://www.alexa.com/siteinfo
Wolk, A. , and Theysohn, S. , 2007, “ Factors Influencing Website Traffic in the Paid Content Market,” J. Mark. Manage., 23(7–8), pp. 769–796. [CrossRef]
Shankar, V. , Urban, G. L. , and Sultan, F. , 2002, “ Online Trust: A Stakeholder Perspective, Concepts, Implications, and Future Directions,” J. Strategic Inf. Syst., 11(3–4), pp. 325–344. [CrossRef]
Shadish, W. , Cook, T. D. , and Campbell, D. T. , 2002, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton Mifflin, Boston, MA.
Bollen, K. A. , and Pearl, J. , 2013, Eight Myths about Causality and Structural Equation Models, Springer, Dordrecht, The Netherlands, pp. 301–328.


Grahic Jump Location
Fig. 1

Interdependence of set of sponsor-related (S), problem-related (P), contest-related (C), individual-related (I), and outcome-related (O) factors in crowdsourcing contests

Grahic Jump Location
Fig. 2

Structure of game-theoretical models

Grahic Jump Location
Fig. 3

Various uncertainties in deciding participant i's probability of winning: 1—Uncertain cost, 2—Uncertainty in design process, 3—Uncertain evaluation criteria, 4—Uncertainty in quality of opponent given by CDFs, and 5—Uncertainty in winning given by CSFs

Grahic Jump Location
Fig. 4

Graph of hypothesized causal relations between influencing and outcome factors, developed based on existing game-theoretic models and empirical studies on crowdsourcing



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In