Research Papers: Design Theory and Methodology

Exploring Biases Between Human and Machine Generated Designs

[+] Author and Article Information
Christian E. Lopez

Industrial and Manufacturing Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: cql5441@psu.edu

Scarlett R. Miller

Engineering Design and Industrial Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: shm13@psu.edu

Conrad S. Tucker

Engineering Design and Industrial Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: ctucker4@psu.edu

1Corresponding author.

2Engineering Design, The Pennsylvania State University, 213 N Hammond Building, State College, PA 16802.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received June 30, 2018; final manuscript received October 26, 2018; published online December 20, 2018. Assoc. Editor: Tahira Reid.

J. Mech. Des 141(2), 021104 (Dec 20, 2018) (10 pages) Paper No: MD-18-1525; doi: 10.1115/1.4041857 History: Received June 30, 2018; Revised October 26, 2018

The objective of this work is to explore the possible biases that individuals may have toward the perceived functionality of machine generated designs, compared to human created designs. Toward this end, 1187 participants were recruited via Amazon mechanical Turk (AMT) to analyze the perceived functional characteristics of both human created two-dimensional (2D) sketches and sketches generated by a deep learning generative model. In addition, a computer simulation was used to test the capability of the sketched ideas to perform their intended function and explore the validity of participants' responses. The results reveal that both participants and computer simulation evaluations were in agreement, indicating that sketches generated via the deep generative design model were more likely to perform their intended function, compared to human created sketches used to train the model. The results also reveal that participants were subject to biases while evaluating the sketches, and their age and domain knowledge were positively correlated with their perceived functionality of sketches. The results provide evidence that supports the capabilities of deep learning generative design tools to generate functional ideas and their potential to assist designers in creative tasks such as ideation.

Copyright © 2019 by ASME
Your Session has timed out. Please sign back in to continue.


Chandrasegaran, S. K. , Ramani, K. , Sriram, R. D. , Horváth, I. , Bernard, A. , Harik, R. F. , and Gao, W. , 2013, “ The Evolution, Challenges, and Future of Knowledge Representation in Product Design Systems,” Comput. Des., 45(2), pp. 204–228. https://www.sciencedirect.com/science/article/pii/S0010448512001741
Liapis, A. , Yannakakis, G. N. , Alexopoulos, C. , and Lopes, P. , 2016, “ Can Computers Foster Human User's Creativity? Theory and Practice of Mixed-Initiative Co-Creativity,” Digital Cult. Educ., 8(2), pp. 136–153. http://www.digitalcultureandeducation.com/uncategorized/liapis-html/
Burnap, A. , Lui, Y. , Pan, Y. , Lee, H. , Gonzalez, R. , and Papalambors, P. , 2016, “ Estimating and Exploring the Product Form Design Space Using Deep Generative Models,” ASME Paper No. DETC2016-60091.
Dering, M. L. , and Tucker, C. S. , 2017, “ Generative Adversarial Networks for Increasing the Veracity of Big Data,” IEEE International Conference on Big Data (BIGDATA), Boston, MA, Dec. 11–14, pp. 2513–2520.
Boden, M. A. , 2004, The Creative Mind: Myths and Mechanisms, 2nd ed., Routledge, New York.
Kazi, R. H. , Grossman, T. , Cheong, H. , Hashemi, A. , and Fitzmaurice, G. , 2017, “ DreamSketch: Early Stage 3D Design Explorations With Sketching and Generative Design,” 30th Annual ACM Symposium on User Interface Software and Technology, Quebec City, QC, Canada, Oct. 22–25, pp. 401–414.
Lopez, C. E. , and Tucker, C. S. , 2018, “ Human Validation of Computer Versus Human Generated Design Sketches,” ASME Paper No. DETC2018-85698.
Rietzschel, E. F. , Nijstad, B. A. , and Stroebe, W. , 2006, “ Productivity is Not Enough: A Comparison of Interactive and Nominal Brainstorming Groups on Idea Generation and Selection,” J. Exp. Soc. Psychol., 42(2), pp. 244–251. [CrossRef]
Toh, C. A. , Strohmetz, A. A. , and Miller, S. R. , 2016, “ The Effects of Gender and Idea Goodness on Ownership Bias in Engineering Design Education,” ASME J. Mech. Des., 138(10), p. 101105. [CrossRef]
Toh, C. A. , Patel, A. H. , Strohmetz, A. A. , and Miller, S. R. , 2015, “ My Idea is Best! Ownership Bias and Its Influence on Engineering Concept Selection,” ASME Paper No. DETC2015-46478.
Zheng, X. , and Miller, S. R. , 2017, “ Risky Business: The Driving Factors of Creative Risk Taking Attitudes in Engineering Design Industry,” ASME Paper No. DETC2017-67799.
Thomson, M. E. , Önkal, D. , Avcioǧlu, A. , and Goodwin, P. , 2004, “ Aviation Risk Perception: A Comparison Between Experts and Novices,” Risk Anal., 24(6), pp. 1585–1595. [CrossRef] [PubMed]
Arning, K. , and Ziefle, M. , 2007, “ Understanding Age Differences in PDA Acceptance and Performance,” Comput. Human Behav., 23(6), pp. 2904–2927. [CrossRef]
Wang, Y.-S. , Wu, M.-C. , and Wang, H.-Y. , 2009, “ Investigating the Determinants and Age and Gender Differences in the Acceptance of Mobile Learning,” Br. J. Educ. Technol., 40(1), pp. 92–118. [CrossRef]
Venkatesh, V. , Morris, M. G. , Davis, G. B. , and Davis, F. D. , 2003, “ User Acceptance of Information Technology: Toward a Unified View,” MIS Q., 27(3), pp. 425–478. [CrossRef]
Orsborn, S. , Cagan, J. , and Boatwright, P. , 2009, “ Quantifying Aesthetic Form Preference in a Utility Function,” ASME J. Mech. Des., 131(6), p. 061001. [CrossRef]
Reid, T. N. , Gonzalez, R. D. , and Papalambros, P. Y. , 2010, “ Quantification of Perceived Environmental Friendliness for Vehicle Silhouette Design,” ASME J. Mech. Des., 132(10), p. 101010. [CrossRef]
Schmidhuber, J. , 2015, “ Deep Learning in Neural Networks: An Overview,” Neural Networks, 61, pp. 85–117. [CrossRef] [PubMed]
Ha, D. , and Eck, D. , 2018, “ A Neural Representation of Sketch Drawings,” Sixth International Conference on Learning Representations, Vancouver, BC, Canada, pp. 1–16.
Chen, Y. , Tu, S. , Yi, Y. , and Xu, L. , 2017, “ Sketch-pix2seq: A Model to Generate Sketches of Multiple Categories,” e-print arXiv:1709.04121. https://arxiv.org/abs/1709.04121
Achlioptas, P. , Diamanti, O. , Mitliagkas, I. , and Guibas, L. , 2017, “ Learning Representations and Generative Models for 3D Point Clouds,” 35th International Conference on Machine Learning, Stockholm, Sweden, July 10–15, pp. 1–20.
Bodén, M. , 2001, “ A Guide to Recurrent Neural Networks and Backpropagation,” Halmstad, Sweden, Dallas Project SICS Technical Report T2002:03.
Goodfellow, I. , Pouget-Abadie, J. , Mirza, M. , Xu, B. , Warde-Farley, D. , Ozair, S. , Courville, A. , and Bengio, Y. , 2014, “ Generative Adversarial Nets,” 27th International Conference on Neural Information Processing Systems (NIPS), Montreal, QC, Canada, Dec. 8–13, pp. 2672–2680.
Dosovitskiy, A. , Springenberg, J. T. , Tatarchenko, M. , and Brox, T. , 2017, “ Learning to Generate Chairs, Tables and Cars With Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell., 39(4), pp. 692–705. [PubMed]
Theis, L. , Oord, A. V. D. , and Bethge, M. , 2016, “ A Note on the Evaluation of Generative Models,” International Conference of Learning Representations, San Juan, Puerto Rico, May 2–4, pp. 1–10.
Poetz, M. K. , and Schreier, M. , 2012, “ The Value of Crowdsourcing: Can Users Really Compete With Professionals in Generating New Product Ideas?,” J. Prod. Innov. Manag., 29(2), pp. 245–256. [CrossRef]
Wang, G. G. , and Shan, S. , 2007, “ Review of Metamodeling Techniques in Support of Engineering Design Optimization,” ASME J. Mech. Des., 129(4), pp. 370–380. [CrossRef]
Dering, M. , and Tucker, C. , 2017, “ A Convolutional Neural Network Model for Predicting a Product's Function, Given Its Form,” ASME J. Mech. Des., 139(11), pp. 1–14. http://mechanicaldesign.asmedigitalcollection.asme.org/article.aspx?articleid=2645713
Buxton, B. , 2010, Sketching User Experiences: Getting the Design Right and the Right Design, Morgan Kaufmann, San Francisco, CA.
Goel, V. , 1997, “ Sketches of Thought,” Des. Stud., 18(1), pp. 129–130. https://www.sciencedirect.com/science/article/pii/S0142694X97832887
Rodgers, P. A. , Green, G. , and McGown, A. , 2000, “ Using Concept Sketches to Track Design Progress,” Des. Stud., 21(5), pp. 451–464. [CrossRef]
Van Der Lugt, R. , 2005, “ How Sketching Can Affect the Idea Generation Process in Design Group Meetings,” Des. Stud., 26(2), pp. 101–112. [CrossRef]
Yang, M. C. , 2009, “ Observations on Concept Generation and Sketching in Engineering Design,” Res. Eng. Des., 20(1), pp. 1–11. [CrossRef]
Macomber, B. , and Yang, M. C. , 2011, “ The Role of Sketch Finish and Style in User Responses to Early Stage Design Concepts,” ASME Paper No. DETC2011-48714.
Häggman, A. , Tsai, G. , Elsen, C. , Honda, T. , and Yang, M. C. , 2015, “ Connections Between the Design Tool, Design Attributes, and User Preferences in Early Stage Design,” ASME J. Mech. Des., 137(7), p. 071101. [CrossRef]
Cunningham, J. , and Tucker, C. S. , 2018, “ A Validation Neural Network (VNN) Metamodel for Predicting the Performance of Deep Generative Designs,” ASME Paper No. DETC2018-86299.
Ren, Y. , Burnap, A. , and Papalambros, P. , 2013, “ Quantification of Perceptual Design Attributes Using a Crowd,” 19th International Conference on Engineering Design, Seoul, South Korea, pp. 19–22.
Toh, C. A. , Miele, L. M. , and Miller, S. R. , 2016, “ Which One Should I Pick? Concept Selection in Engineering Design Industry,” ASME Paper No. DETC2015-46522.
Cox, D. , and Cox, A. D. , 2002, “ Beyond First Impressions: The Effects of Repeated Exposure on Consumer Liking of Visually Complex and Simple Product Designs,” J. Acad. Mark. Sci., 30(2), pp. 119–130. [CrossRef]
Mueller, J. S. , Melwani, S. , and Goncalo, J. A. , 2012, “ The Bias Against Creativity: Why People Desire but Reject Creative Ideas,” Psychol. Sci., 23(1), pp. 13–17. [CrossRef] [PubMed]
Toh, C. A. , and Miller, S. R. , 2014, “ The Role of Individual Risk Attitudes on the Selection of Creative Concepts in Engineering Design,” ASME Paper No. DETC2014-35106.
Burnap, A. , Gerth, R. , Gonzalez, R. , and Papalambros, P. Y. , 2017, “ Identifying Experts in the Crowd for Evaluation of Engineering Designs,” J. Eng. Des., 28(5), pp. 317–337. [CrossRef]
W. , I. J., Nap , H. H., De Kort , Y. , and Poels, K. , 2007, “ Digital Game Design for Elderly Users,” Conference on Future Play, Toronto, ON, Canada, Nov. 14–17, pp. 17–22.
Parasuraman, R. , and Manzey, D. H. , 2010, “ Complacency and Bias in Human Use of Automation: An Attentional Integration,” Hum. Factors, 52(3), pp. 381–410. [CrossRef] [PubMed]
Mosier, K. L. , and Skitka, L. J. , 1996, “ Human Decision Makers and Automated Decision Aids: Made for Each Other?,” Automation and Human Performance, Erlbaum, Hillsdale, NJ.
Lee, J. D. , and See, K. A. , 2004, “ Trust in Automation: Designing for Appropriate Reliance,” Hum. Factors, 46(1), pp. 50–80. http://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392 [PubMed]
Dzindolet, M. T. , Pierce, L. G. , Beck, H. P. , and Dawe, L. A. , 2002, “ The Perceived Utility of Human and Automated Aids in a Visual Detection Task,” Hum. Factors, 44(1), pp. 79–94. [CrossRef] [PubMed]
Le, Q. , and Panchal, J. H. , 2011, “ Modeling the Effect of Product Architecture on Mass-Collaborative Processes,” ASME J. Comput. Inf. Sci. Eng., 11(1), p. 011003. http://computingengineering.asmedigitalcollection.asme.org/article.aspx?articleid=1402261
Jongejan, J. , Rowley, H. , Kawashima, T. , Kim, J. , and Fox-Gieg, N. , 2016, “ The Quick, Draw!—A.I. Experiment,” Mount View, CA, accessed Feb. 17, 2018, https://quickdraw.withgoogle.com/
Buchanan, T. , 2000, Psychological Experiments on the Internet, Academic Press, Oxford, UK.
Mason, W. , and Suri, S. , 2012, “ Conducting Behavioral Research on Amazon's Mechanical Turk,” Behav. Res. Methods, 44(1), pp. 1–23. https://link.springer.com/article/10.3758/s13428-011-0124-6 [PubMed]
Unity, 2017, “ Unity—Game Engine,” San Francisco, CA, accessed May 31, 2018, https://www.unity3d.com
González, J. D. , Escobar, J. H. , Sánchez, H. , De La Hoz, J. , and Beltrán, J. R. , 2017, “ 2D and 3D Virtual Interactive Laboratories of Physics on Unity Platform,” J. Phys.: Conf. Ser., 935(1), p. 012069. http://iopscience.iop.org/article/10.1088/1742-6596/935/1/012069
Ballu, A. , Yan, X. , Blanchard, A. , Clet, T. , Mouton, S. , and Niandou, H. , 2016, “ Virtual Metrology Laboratory for e-Learning,” Procedia CIRP, 43(1), pp. 148–153. https://www.sciencedirect.com/science/article/pii/S2212827116003929
Cortina, J. M. , 1993, “ What is Coefficient Alpha? an Examination of Theory and Applications,” J. Appl. Psychol., 78(1), pp. 98–104. http://psycnet.apa.org/record/1993-19965-001
Cohen, J. , 1988, Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Routledge, New York.


Grahic Jump Location
Fig. 1

Example of human and computer generated boat sketches

Grahic Jump Location
Fig. 2

Instruction page from questionnaire

Grahic Jump Location
Fig. 3

Computer simulation environment in Unity



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In