Research Papers: Design Theory and Methodology

Automatic Facial Feature Extraction for Predicting Designers' Comfort With Engineering Equipment During Prototype Creation

[+] Author and Article Information
Shruthi Bezawada

Industrial and Manufacturing Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: srb321@psu.edu

Qianyu Hu

Industrial and Manufacturing Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: qzh5042@psu.edu

Allison Gray

Human Development and Family Studies,
The Pennsylvania State University,
State College, PA 16802
e-mail: axg5562@psu.edu

Timothy Brick

Human Development and Family Studies,
The Pennsylvania State University,
State College, PA 16802
e-mail: tbrick@psu.edu

Conrad Tucker

Industrial and Manufacturing Engineering,
The Pennsylvania State University,
State College, PA 16802
e-mail: ctucker4@psu.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received April 15, 2016; final manuscript received November 25, 2016; published online January 6, 2017. Assoc. Editor: Andy Dong.

J. Mech. Des 139(2), 021102 (Jan 06, 2017) (10 pages) Paper No: MD-16-1285; doi: 10.1115/1.4035428 History: Received April 15, 2016; Revised November 25, 2016

Designers frequently utilize engineering equipment to create physical prototypes during the iterative concept generation and prototyping phases of design. Currently, evaluating designers' efficiency during prototype creation is a manual process that either involves observational or survey based approaches. Real-time feedback when using engineering equipment has the potential to enhance designers' efficiency or mitigate potential injuries that may result from incorrect use of equipment. Toward an automated approach to addressing these challenges, the authors of this work test the hypotheses that (i) there exists a difference in designers' comfort levels before and after they use a piece of engineering prototyping equipment and (ii) a machine learning model predicts the level of comfort a designer has while using engineering prototyping equipment with accuracies greater than random chance. It has been shown that the level of comfort that an individual has while completing a task impacts their performance. The authors investigate whether automatic tracking of designers' facial expressions during prototype creation predicts their level of comfort. A study, involving 37 participants using various engineering equipment, is used to validate the approach. The support vector machine (SVM) regression model yielded a range of R squared values from 0.82 to 0.86 for an equipment-specific model. A general model built to predict comfort level across all engineering equipment yielded an R squared value of 0.68. This work has the potential to transform the manner in which design teams utilize engineering equipment toward more efficient concept generation and prototype creation processes.

Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.


Cinnéide, M. Ó. , and Nixon, P. , 1999, “ A Methodology for the Automated Introduction of Design Patterns,” 1999 IEEE International Conference on Software Maintenance (ICSM’99), Aug. 30–Sept. 2, pp. 463–472.
Römer, A. , Pache, M. , Weißhahn, G. , Lindemann, U. , and Hacker, W. , 2001, “ Effort-Saving Product Representations in Design—Results of a Questionnaire Survey,” Des. Stud., 22(6), pp. 473–491. [CrossRef]
Camburn, B. , Dunlap, B. , Gurjar, T. , Hamon, C. , Green, M. , Jensen, D. , Crawford, R. , Otto, K. , and Wood, K. , 2015, “ A Systematic Method for Design Prototyping,” ASME J. Mech. Des., 137(8), p. 81102. [CrossRef]
Lim, Y.-K. , Stolterman, E. , and Tenenberg, J. , 2008, “ The Anatomy of Prototypes: Prototypes as Filters, Prototypes as Manifestations of Design Ideas,” ACM Trans. Comput.-Hum. Interact. (TOCHI), 15(2), p. 7. [CrossRef]
Zuboff, S. , 1988, In the Age of the Smart Machine: The Future of Work and Power, Basic Books, New York.
Dul, J. , and Neumann, W. P. , 2009, “ Ergonomics Contributions to Company Strategies,” Appl. Ergon., 40(4), pp. 745–752. [CrossRef] [PubMed]
Teizer, J. , Venugopal, M. , and Walia, A. , 2008, “ Ultrawideband for Automated Real-Time Three-Dimensional Location Sensing for Workforce, Equipment, and Material Positioning and Tracking,” Transp. Res. Rec.: J. Transp. Res. Board, 2081, pp. 56–64. [CrossRef]
Eppinger, S. D. , and Whitney, D. E. , 1995, “ Accelerating Product Development by the Exchange of Preliminary Product Design Information,” ASME J. Mech. Des., 117(4), pp. 491–498. [CrossRef]
Field, B. W. , 2007, “ Visualization, Intuition, and Mathematics Metrics as Predictors of Undergraduate Engineering Design Performance,” ASME J. Mech. Des., 129(7), pp. 735–743. [CrossRef]
Bordegoni, M. , Cugini, U. , Caruso, G. , and Polistina, S. , 2009, “ Mixed Prototyping for Product Assessment: A Reference Framework,” Int. J. Interact. Des. Manuf. (IJIDeM), 3(3), pp. 177–187. [CrossRef]
Dong, A. , Hill, A. W. , and Agogino, A. M. , 2004, “ A Document Analysis Method for Characterizing Design Team Performance,” ASME J. Mech. Des., 126(3), pp. 378–385. [CrossRef]
Stempfle, J. , and Badke-Schaub, P. , 2002, “ Thinking in Design Teams-An Analysis of Team Communication,” Des. Stud., 23(5), pp. 473–496. [CrossRef]
Behoora, I. , and Tucker, C. S. , 2015, “ Machine Learning Classification of Design Team Members' Body Language Patterns for Real Time Emotional State Detection,” Des. Stud., 39, pp. 100–127. [CrossRef]
Munoz, D. , and Tucker, C. S. , 2016, “ Modeling the Semantic Structure of Textually-Derived Learning Content and Its Impact on Recipients Response States,” ASME J. Mech. Des., 138(4), p. 042001. [CrossRef]
Frith, C. , 2009, “ Role of Facial Expressions in Social Interactions,” Philos. Trans. R. Soc., B, 364(1535), pp. 3453–3458. [CrossRef]
Kapoor, A. , and Picard, R. W. , 2005, “ Multimodal Affect Recognition in Learning Environments,” 13th Annual ACM International Conference on Multimedia, Singapore, Nov. 6–11, pp. 677–682.
Calvo, R. A. , and D'Mello, S. , 2010, “ Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications,” IEEE Trans. Affective Comput., 1(1), pp. 18–37. [CrossRef]
Linsey, J. S. , Tseng, I. , Fu, K. , Cagan, J. , Wood, K. L. , and Schunn, C. , 2010, “ A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty,” ASME J. Mech. Des., 132(4), p. 041003. [CrossRef]
Saragih, J. M. , Lucey, S. , and Cohn, J. F. , 2010, “ Deformable Model Fitting by Regularized Landmark Mean-Shift,” Int. J. Comput. Vision, 91(2), pp. 200–215. [CrossRef]
Wang, L. , Shen, W. , Xie, H. , Neelamkavil, J. , and Pardasani, A. , 2002, “ Collaborative Conceptual Design—State of the Art and Future Trends,” Comput.-Aided Des., 34(13), pp. 981–996. [CrossRef]
Powers, M. F. , and Jones-Walker, J. , 2005, “ An Interdisciplinary Collaboration to Improve Critical Thinking Among Pharmacy Students,” Am. J. Pharm. Educ., 69(1–5), p. 516.
Hutchinson, H. , Mackay, W. , Westerlund, B. , Bederson, B. B. , Druin, A. , Plaisant, C. , Beaudouin-Lafon, M. , Conversy, S. , Evans, H. , and Hansen, H. , 2003, “ Technology Probes: Inspiring Design for and With Families,” SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, Apr. 5–10, pp. 17–24.
Yang, M. C. , 2005, “ A Study of Prototypes, Design Activity, and Design Outcome,” Des. Stud., 26(6), pp. 649–669. [CrossRef]
Gero, J. S. , 1990, “ Design Prototypes: A Knowledge Representation Schema for Design,” AI Mag., 11(4), p. 26.
Qian, L. , and Gero, J. S. , 1996, “ Function–Behavior–Structure Paths and Their Role in Analogy-Based Design,” Artif. Intell. Eng., Des., Anal. Manuf., 10(4), pp. 289–312. [CrossRef]
Gerber, E. , and Carroll, M. , 2012, “ The Psychological Experience of Prototyping,” Des. Stud., 33(1), pp. 64–84. [CrossRef]
Fadier, E. , and De la Garza, C. , 2006, “ Safety Design: Towards a New Philosophy,” Saf. Sci., 44(1), pp. 55–73. [CrossRef]
Pavlovic-Veselinovic, S. , 2014, “ Ergonomics as a Missing Part of Sustainability,” Work, 49(3), pp. 395–399. [PubMed]
Jang, J. , and Schunn, C. D. , 2012, “ Physical Design Tools Support and Hinder Innovative Engineering Design,” ASME J. Mech. Des., 134(4), p. 41001. [CrossRef]
Busby, J. S. , 1998, “ The Neglect of Feedback in Engineering Design Organisations,” Des. Stud., 19(1), pp. 103–117. [CrossRef]
Booth, J. W. , Taborda, E. A. , Ramani, K. , and Reid, T. , 2016, “ Interventions for Teaching Sketching Skills and Reducing Inhibition for Novice Engineering Designers,” Des. Stud., 43, pp. 1–23. [CrossRef]
Busseri, M. A. , and Palmer, J. M. , 2000, “ Improving Teamwork: The Effect of Self-Assessment on Construction Design Teams,” Des. Stud., 21(3), pp. 223–238. [CrossRef]
Adams, R. S. , Turns, J. , and Atman, C. J. , 2003, “ Educating Effective Engineering Designers: The Role of Reflective Practice,” Des. Stud., 24(3), pp. 275–294. [CrossRef]
Austin-Breneman, J. , Honda, T. , and Yang, M. C. , 2012, “ A Study of Student Design Team Behaviors in Complex System Design,” ASME J. Mech. Des., 134(12), p. 124504. [CrossRef]
Lauche, K. , 2005, “ Job Design for Good Design Practice,” Des. Stud., 26(2), pp. 191–213. [CrossRef]
Bernal, M. , Haymaker, J. R. , and Eastman, C. , 2015, “ On the Role of Computational Support for Designers in Action,” Des. Stud., 41, pp. 163–182. [CrossRef]
Dinar, M. , Shah, J. J. , Cagan, J. , Leifer, L. , Linsey, J. , Smith, S. M. , and Hernandez, N. V. , 2015, “ Empirical Studies of Designer Thinking: Past, Present, and Future,” ASME J. Mech. Des., 137(2), p. 21101. [CrossRef]
McDuff, D. , El Kaliouby, R. , Senechal, T. , Demirdjian, D. , and Picard, R. , 2014, “ Automatic Measurement of Ad Preferences From Facial Responses Gathered Over the Internet,” Image Vision Comput., 32(10), pp. 630–640. [CrossRef]
Ekman, P. , 1992, “ An Argument for Basic Emotions,” Cognit. Emotion, 6(3–4), pp. 169–200. [CrossRef]
Pantic, M. , and Rothkrantz, L. J. , 2000, “ Automatic Analysis of Facial Expressions: The State of the Art,” IEEE Trans. Pattern Anal. Mach. Intell., 22(12), pp. 1424–1445. [CrossRef]
Zeng, Z. , Tu, J. , Liu, M. , Zhang, T. , Rizzolo, N. , Zhang, Z. , Huang, T. S. , Roth, D. , and Levinson, S. , 2004, “ Bimodal HCI-Related Affect Recognition,” 6th International Conference on Multimodal Interfaces, State College, PA, Oct. 13–15, pp. 137–143.
Khan, F. A. , Weippl, E. R. , and Tjoa, A. M. , 2009, “ Integrated Approach for the Detection of Learning Styles and Affective States,” World Conference on Educational Multimedia, Hypermedia and Telecommunications, pp. 753–761.
Dimitriadou, A. E. , Hornik, K. , Leisch, F. , Meyer, D. , Weingessel, A. , and Leisch, M. F. , 2006, “ The e1071 Package, Misc. Functions of Department of Statistics (e1071),” TU Wien, Wien Austria.
Block, J. , 1995, “ Going Beyond the Five Factors Given: Rejoinder to Costa and McCrae (1995) and Goldberg and Saucier (1995),” Psychol. Bull., 117(2), pp. 226–229. [CrossRef]
Meyer, D. , Dimitriadou, E. , Hornik, K. , Weingessel, A. , and Leisch, F. , 2014, “ E1071: Misc. Functions of the Department of Statistics (e1071),” R package version 1.6-2, TU Wien, Wien, Austria.
Sakia, R. M. , 1992, “ The Box-Cox Transformation Technique: A Review,” The Statistician, 41(2), pp. 169–178. [CrossRef]


Grahic Jump Location
Fig. 4

Procrustes analysis: the left figure is the pre-aligned figure and the right figure is the aligned figure

Grahic Jump Location
Fig. 3

Extraction of facial key points from video sequences

Grahic Jump Location
Fig. 2

A bandsaw being used to cut material in an engineering workspace

Grahic Jump Location
Fig. 1

Overview of method

Grahic Jump Location
Fig. 5

One-dimensional nonlinear regression with epsilon intensive band

Grahic Jump Location
Fig. 6

Experimental layout

Grahic Jump Location
Fig. 7

Tasks to be performed with the power saw as well as scissors stations

Grahic Jump Location
Fig. 8

Tasks to be performed at the drill station

Grahic Jump Location
Fig. 9

Snapshot of video sequence

Grahic Jump Location
Fig. 10

Conversion of raw facial key point data into data ready for analysis

Grahic Jump Location
Fig. 11

Statistical summary of level of comfort at the three workstations



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In