0
Research Papers: Design Theory and Methodology

Capturing Human Sequence-Learning Abilities in Configuration Design Tasks Through Markov Chains

[+] Author and Article Information
Christopher McComb

Mem. ASME
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: ccm@cmu.edu

Jonathan Cagan

Fellow ASME
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: cagan@cmu.edu

Kenneth Kotovsky

Department of Psychology,
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: kotovsky@cmu.edu

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received November 14, 2016; final manuscript received June 16, 2017; published online July 12, 2017. Assoc. Editor: Katja Holtta-Otto.

J. Mech. Des 139(9), 091101 (Jul 12, 2017) (12 pages) Paper No: MD-16-1761; doi: 10.1115/1.4037185 History: Received November 14, 2016; Revised June 16, 2017

Designers often search for new solutions by iteratively adapting a current design. By engaging in this search, designers not only improve solution quality but also begin to learn what operational patterns might improve the solution in future iterations. Previous work in psychology has demonstrated that humans can fluently and adeptly learn short operational sequences that aid problem-solving. This paper explores how designers learn and employ sequences within the realm of engineering design. Specifically, this work analyzes behavioral patterns in two human studies in which participants solved configuration design problems. Behavioral data from the two studies are first analyzed using Markov chains to determine how much representation complexity is necessary to quantify the sequential patterns that designers employ during solving. It is discovered that first-order Markov chains are capable of accurately representing designers' sequences. Next, the ability to learn first-order sequences is implemented in an agent-based modeling framework to assess the performance implications of sequence-learning abilities. These computational studies confirm the assumption that the ability to learn sequences is beneficial to designers.

FIGURES IN THIS ARTICLE
<>
Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.

References

McComb, C. , Cagan, J. , and Kotovsky, K. , 2015, “ Rolling With the Punches: An Examination of Team Performance in a Design Task Subject to Drastic Changes,” Des. Studies, 36(1), pp. 99–121. [CrossRef]
McComb, C. , Cagan, J. , and Kotovsky, K. , 2015, “ Lifting the Veil: Drawing Insights About Design Teams From a Cognitively-Inspired Computational Model,” Des. Studies, 40, pp. 119–142. [CrossRef]
Neill, T. M. , Gero, J. S. , and Warren, J. , 1998, “ Understanding Conceptual Electronic Design Using Protocol Analysis,” Res. Eng. Des., 10(3), pp. 129–140. [CrossRef]
Kan, J. W. , and Gero, J. S. , 2011, “ Comparing Designing Across Different Domains: An Exploratory Case Study,” 18th International Conference on Engineering Design (ICED), Lyngby/Copenhagen, Denmark, Aug. 15–19, pp. 194–203. https://www.designsociety.org/publication/30470/comparing_designing_across_different_domains_an_exploratory_case_study
Yu, R. , and Gero, J. S. , 2015, “ An Empirical Foundation for Design Patterns in Parametric Design,” 20th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Daegu, South Korea, May 20–23, pp. 1–9. http://nova.newcastle.edu.au/vital/access/manager/Repository/uon:26697
Gero, J. S. , and Peng, W. , 2009, “ Understanding Behaviors of a Constructive Memory Agent: A Markov Chain Analysis,” Knowl. Based Syst., 22(8), pp. 610–621. [CrossRef]
March, J. G. , 1991, “ Exploration and Exploitation in Organizational Learning,” Organ. Sci., 2(1), pp. 71–87. [CrossRef]
Gupta, A. K. , Smith, K. G. , and Shalley, C. E. , 2006, “ The Interplay Between Exploration and Exploitation,” Acad. Manage. J., 49(4), pp. 693–706. [CrossRef]
Goncher, A. , Johri, A. , Kothaneth, S. , and Lohani, V. , 2009, “ Exploration and Exploitation in Engineering Design: Examining the Effects of Prior Knowledge on Creativity and Ideation,” 39th IEEE Frontiers in Education Conference (FIE), San Antonio, TX, Oct. 18–21, pp. 1–7.
Kotovsky, K. , Hayes, J. , and Simon, H. A. , 1985, “ Why Are Some Problems Hard? Evidence From Tower of Hanoi,” Cognit. Psychol., 17(2), pp. 248–294. [CrossRef]
Kotovsky, K. , and Simon, H. A. , 1990, “ What Makes Some Problems Really Hard: Explorations in the Problem Space of Difficulty,” Cognit. Psychol., 22(2), pp. 143–183. [CrossRef]
Nissen, M. J. , and Bullemer, P. , 1987, “ Attentional Requirements of Learning: Evidence From Performance Measures,” Cognit. Psychol., 19(1), pp. 1–32. [CrossRef]
Reed, J. , and Johnson, P. , 1994, “ Assessing Implicit Learning With Indirect Tests: Determining What Is Learned About Sequence Structure,” J. Exp. Psychol., 20(3), pp. 585–594.
Clegg, B. A. , Digirolamo, G. J. , and Keele, S. W. , 1998, “ Sequence Learning,” Trends Cognit. Sci., 2(8), pp. 275–281. [CrossRef]
Chase, W. G. , and Simon, H. A. , 1973, “ Perception in Chess,” Cognit. Psychol., 4(1), pp. 55–81. [CrossRef]
Egan, D. E. , and Schwartz, B. J. , 1979, “ Chunking in Recall of Symbolic Drawings,” Mem. Cognit., 7(2), pp. 149–158. [CrossRef] [PubMed]
Reitman, J. S. , 1976, “ Skilled Perception in Go: Deducing Memory Structures From Inter-Response Times,” Cognit. Psychol., 8(3), pp. 336–356. [CrossRef]
Moss, J. , Cagan, J. , and Kotovsky, K. , 2004, “ Learning From Design Experience in an Agent-Based Design System,” Res. Eng. Des., 15(2), pp. 77–92.
Pretz, J. E. , Naples, A. J. , and Sternberg, R. J. , 2003, “ Recognizing, Defining, and Representing Problems,” The Psychology of Problem Solving, J. E. Davidson , and R. J. Sternberg , eds., Cambridge University Press, New York.
Simon, H. A. , and Kotovsky, K. , 1963, “ Human Acquisition of Concepts for Sequential Patterns,” Psychol. Rev., 70(6), pp. 534–546. [CrossRef] [PubMed]
Kotovsky, K. , and Simon, H. A. , 1973, “ Empirical Tests of a Theory of Human Acquisition of Concepts for Sequential Patterns,” Cognit. Psychol., 4(3), pp. 399–424. [CrossRef]
Novick, L. R. , and Tversky, B. , 1987, “ Cognitive Constraints on Ordering Operations: The Case of Geometric Analogies,” J. Exp. Psychol.: Gen., 116(1), pp. 50–67. [CrossRef] [PubMed]
Perruchet, P. , and Amorim, M.-A. , 1992, “ Conscious Knowledge and Changes in Performance in Sequence Learning: Evidence Against Dissociation,” J. Exp. Psychol., 18(4), pp. 785–800. https://www.ncbi.nlm.nih.gov/pubmed/1385616
Willingham, D. B. , Nissen, M. J. , and Bullemer, P. , 1989, “ On the Development of Procedural Knowledge,” J. Exp. Psychol., 15(6), pp. 1047–1060. https://www.ncbi.nlm.nih.gov/pubmed/2530305
Curran, T. , and Keele, S. W. , 1993, “ Attentional and Nonattentional Forms of Sequence Learning,” J. Exp. Psychol., 19(1), pp. 189–202. http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&uid=1993-16334-001
Stempfle, J. , and Badke-schaub, P. , 2002, “ Thinking in Design Teams—An Analysis of Team Communication,” Des. Studies, 23(5), pp. 473–496. [CrossRef]
Atman, C. J. , Adams, R. S. , Cardella, M. E. , Turns, J. , Mosborg, S. , and Saleem, J. , 2007, “ Engineering Design Processes: A Comparison of Students and Expert Practitioners,” J. Eng. Educ., 96(4), pp. 359–379. [CrossRef]
Goldschmidt, G. , and Rodgers, P. A. , 2013, “ The Design Thinking Approaches of Three Different Groups of Designers Based on Self-Reports,” Des. Studies, 34(4), pp. 454–471. [CrossRef]
Radcliffe, D. F. , and Lee, T. Y. , 1989, “ Design Methods Used by Undergraduate Engineering Students,” Des. Studies, 10(4), pp. 199–207. [CrossRef]
Todd, D. , 1997, “ Multiple Criteria Genetic Algorithms in Engineering Design and Operation,” Ph.D. thesis, University of Newcastle, Newcastle-upon-Tyne, UK.
Rogers, J. , 1996, “ DeMAID/GA—An Enhanced Design Manager's Aid for Intelligent Decomposition,” AIAA Paper No. 96-4157.
Sen, C. , Ameri, F. , and Summers, J. D. , 2010, “ An Entropic Method for Sequencing Discrete Design Decisions,” ASME J. Mech. Des., 132(10), p. 101004. [CrossRef]
Waldron, M. B. , and Waldron, K. J. , 1988, “ A Time Sequence Study of a Complex Mechanical System Design,” Des. Studies, 9(2), pp. 95–106. [CrossRef]
Meier, C. , Yassine, A. A. , and Browning, T. R. , 2007, “ Design Process Sequencing With Competent Genetic Algorithms,” ASME J. Mech. Des., 129(6), pp. 566–585. [CrossRef]
Gero, J. S. , 1990, “ Design Prototypes: A Knowledge Representation Schema for Design,” AI Mag., 11(4), pp. 26–36. https://www.aaai.org/ojs/index.php/aimagazine/article/view/854
Bhatta, S. R. , and Goel, A. K. , 1992, “ Discovery of Physical Principles From Design Experiences,” International Machine Learning Workshop, San Mateo, CA [Int. J. AI EDAM 8(2), pp. 1–22 (1994)]. https://pdfs.semanticscholar.org/fc0f/467ab4ead20a0a0771a2c8ab9fb102f14b1c.pdf
Gero, J. S. , Kan, J. W. , and Pourmohamadi, M. , 2011, “ Analysing Design Protocols: Development of Methods and Tools,” Research Into Design, Research Publishing, Singapore, pp. 3–10.
Vale, C. A. W. , and Shea, K. , 2003, “ A Machine Learning-Based Approach to Accelerating Computational Design Synthesis,” 14th International Conference on Engineering Design (ICED), Stockholm, Sweden, Aug. 19–21, pp. 183–184. https://www.designsociety.org/publication/24114/a_machine_learning-based_approach_to_accelerating_computational_design_synthesis
Stroock, D. W. , 2005, An Introduction to Markov Processes, Springer Science and Business Media, Berlin.
Markov, A. A. , 1907, “ Extension of the Limit Theorems of Probability Theory to a Sum of Variables Connected in a Chain,” The Notes of the Imperial Academy of Sciences of St. Petersburg, VIII Series, Physio-Mathematical College XXII, Imperial Academy of Science, St. Petersburg, Russia.
Scherr, A. L. , 1962, An Analysis of Time-Shared Computer Systems, Massachusetts Institute of Technology, Cambridge, MA.
Page, L. , Brin, S. , Motwani, R. , and Winograd, T. , 1999, “ The PageRank Citation Ranking: Bringing Order to the Web,” Stanford InfoLab, Stanford, CA, Technical Report No. 422 http://ilpubs.stanford.edu:8090/422/.
Tamir, A. , 1998, Applications of Markov Chains in Chemical Engineering, Elsevier, Amsterdam, The Netherlands.
Gero, J. S. , and Kan, J. W. , 2009, “ Learning to Collaborate During Team Designing: Some Preliminary Results From Measurement-Based Tools,” Third International Conference on Research Into Design Engineering (ICORD), Bangalore, India, Jan. 10–12, pp. 560–567. http://www.academia.edu/27869106/Learning_to_collaborate_during_team_designing_Some_preliminary_results_from_measurement-based_tools
Gero, J. S. , and Kan, J. W. , 2011, “ Learning to Collaborate During Team Designing: Quantitative Measurement,” Third International Conference on Research Into Design Engineering (ICORD), Bangalore, India, Jan. 10–12, pp. 978–981. http://mason.gmu.edu/~jgero/publications/2011/11KanGeroCoRD11.pdf
Raftery, A. E. , 1985, “ A Model for High-Order Markov Chains,” J. R. Stat. Soc. Ser. B, 47(3), pp. 528–539. http://www.jstor.org/stable/2345788
McComb, C. , Cagan, J. , and Kotovsky, K. , 2017, “ Optimizing Design Teams Based on Problem Properties: Computational Team Simulations and an Applied Empirical Test,” ASME J. Mech. Des., 139(4), p. 041101. [CrossRef]
Arlot, S. , and Celisse, A. , 2010, “ A Survey of Cross-Validation Procedures for Model Selection,” Stat. Surv., 4, pp. 40–79. [CrossRef]
Stone, M. , 1974, “ Cross-Validatory Choice and Assessment of Statistical Predictions,” J. R. Stat. Soc. Ser. B, 36(2), pp. 111–147.
Hastie, T. , Tibshirani, R. , and Friedman, J. , 2009, The Elements of Statistical Learning, Springer, New York.
Tversky, A. , and Kahneman, D. , 1973, “ Availability: A Heuristic for Judging Frequency and Probability,” Cognit. Psychol., 5(2), pp. 207–232. [CrossRef]
Bishop, C. M. , and Lasserre, J. , 2007, “ Generative or Discriminative? Getting the Best of Both Worlds,” Bayesian Statistics, J. M. Bernardo , M. J. Bayarri , J. O. Berger , A. P. Dawid , D. Heckerman , A. F. M. Smith , and M. West , eds., Oxford University Press, Oxford, UK, pp. 3–24.
Bellman, R. , 1957, “ A Markovian Decision Process,” J. Math. Mech., 6(5), pp. 679–684. http://www.jstor.org/stable/24900506
Chi, M. T. H. , Glaser, R. , and Rees, E. , 1982, “ Expertise in Problem Solving,” Advances in the Psychology of Human Intelligence, Erlbaum, Hillsdale, NJ, pp. 7–75.
Dinar, M. , Park, Y.-S. , Shah, J. J. , and Langley, P. , 2015, “ Patterns of Creative Design: Predicting Ideation From Problem Formulation,” ASME Paper No. DETC2015-46537.
McComb, C. , Cagan, J. , and Kotovsky, K. , 2016, “ Utilizing Markov Chains to Understand Operation Sequencing in Design Tasks,” Design Computing and Cognition’16, J. S. Gero , ed., Springer International Publishing, Cham, Switzerland, pp. 401–418.

Figures

Grahic Jump Location
Fig. 1

Example Markov chain with three states

Grahic Jump Location
Fig. 2

Example truss operation sequences, with numbers corresponding to the list of truss operations

Grahic Jump Location
Fig. 3

Example cooling system operation sequence, with numbers corresponding to the list of system operations and shading indicating room temperatures

Grahic Jump Location
Fig. 4

Summary of results for truss design study, including (a) log-likelihood computed for models of increasing accuracy, (b) transition matrix of first-order Markov model, and (c) state frequencies for zero-order Markov model

Grahic Jump Location
Fig. 5

Thresholded graph-based visualization of the first-order Markov model for the truss design study. Transition probabilities are depicted by the line thickness (or circle edge thickness for self-transitions).

Grahic Jump Location
Fig. 6

Exemplar sequences extracted from the first-order Markov model for truss design study

Grahic Jump Location
Fig. 7

Summary of results for cooling system design study, including (a) log-likelihood computed for models of increasing accuracy, (b) transition matrix of first-order Markov model, and (c) state frequencies for zero-order Markov model

Grahic Jump Location
Fig. 8

Thresholded graph-based visualization of the first-order Markov model for the cooling system design study. Transition probabilities are depicted by the line thickness (or circle edge thickness for self-transitions).

Grahic Jump Location
Fig. 9

Exemplar sequences extracted from the first-order Markov model for cooling system design study

Grahic Jump Location
Fig. 10

Comparison of truss design quality for zero-order Markov model, first-order Markov model, and human performance on the truss design problem (error bars show ±1 SE)

Grahic Jump Location
Fig. 11

Comparison of cooling system design quality for zero-order Markov model, first-order Markov model, and human performance on the cooling system design problem (error bars show ±1 SE)

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In