Research Papers: Design Theory and Methodology

Gaming the System: An Agent-Based Model of Estimation Strategies and their Effects on System Performance

[+] Author and Article Information
John Meluso

Global Design Laboratory,
Design Science Program,
University of Michigan,
Ann Arbor, MI 48109
e-mail: jmeluso@umich.edu

Jesse Austin-Breneman

Global Design Laboratory,
Department of Mechanical Engineering,
University of Michigan,
Ann Arbor, MI 48109
e-mail: jausbren@umich.edu

1Corresponding author.

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received August 9, 2017; final manuscript received January 30, 2018; published online September 18, 2018. Assoc. Editor: Katja Holtta-Otto.

J. Mech. Des 140(12), 121101 (Sep 18, 2018) (9 pages) Paper No: MD-17-1548; doi: 10.1115/1.4039494 History: Received August 09, 2017; Revised January 30, 2018

Parameter estimates in large-scale complex engineered systems (LaCES) affect system evolution, yet can be difficult and expensive to test. Systems engineering uses analytical methods to reduce uncertainty, but a growing body of work from other disciplines indicates that cognitive heuristics also affect decision-making. Results from interviews with expert aerospace practitioners suggest that engineers bias estimation strategies. Practitioners reaffirmed known system features and posited that engineers may bias estimation methods as a negotiation and resource conservation strategy. Specifically, participants reported that some systems engineers “game the system” by biasing requirements to counteract subsystem estimation biases. An agent-based model (ABM) simulation which recreates these characteristics is presented. Model results suggest that system-level estimate accuracy and uncertainty depend on subsystem behavior and are not significantly affected by systems engineers' “gaming” strategy.

Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.


Bloebaum, C. L. , and McGowan, A.-M. R. , 2012, “ The Design of Large-Scale Complex Engineered Systems: Present Challenges and Future Promise,” AIAA Paper No. 2012-5571.
McGowan, A.-M. R. , Daly, S. , Baker, W. , Papalambros, P. , and Seifert, C. , 2013, “ A Socio-Technical Perspective on Interdisciplinary Interactions During the Development of Complex Engineered Systems,” Procedia Comput. Sci., 16, pp. 1142–1151. [CrossRef]
McGowan, A.-M. R. , 2014, “ Interdisciplinary Interactions During R & D and Early Design of Large Engineered Systems,” Ph.D. thesis, University of Michigan, Ann Arbor, MI. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20140004579.pdf
Yassine, A. , and Braha, D. , 2003, “ Complex Concurrent Engineering and the Design Structure Matrix Method,” Concurrent Eng. Res. Appl., 11(3), pp. 165–176. [CrossRef]
Eppinger, S. , 1991, “ Model-Based Approaches to Managing Concurrent Engineering,” J. Eng. Des., 2(4), pp. 283–290. [CrossRef]
Ye, Y. , Jankovic, M. , and Kremer, G. E. , 2015, “ Understanding the Impact of Subjective Uncertainty on Architecture and Supplier Identification in Early Complex Systems Design,” ASME J. Risk Uncertainty Part B, 1(3), p. 031005. [CrossRef]
de Weck, O. L. , Eckert, C. M. , and Clarkson, J. , 2007, “ A Classification of Uncertainty for Early Product and System Design,” Guidelines for a Decision Support Method Adapted to NPD Processes, Massachusetts Institute of Technology, Cambridge, MA.
Crossland, R. , Williams, J. H. S. , and McMahon, C. A. , 2003, “ An Object-Oriented Modeling Framework for Representing Uncertainty in Early Variant Design,” Res. Eng. Des., 14(3), pp. 173–183. [CrossRef]
Dubos, G. F. , Castet, J. F. , and Saleh, J. H. , 2010, “ Statistical Reliability Analysis of Satellites by Mass Category: Does Spacecraft Size Matter?” Acta Astronaut., 67(5–6), pp. 584–595. [CrossRef]
Ryan, R. S. , and Townsend, J. S. , 1997, “ Fundamentals and Issues in Launch Vehicle Design,” J. Spacecr. Rockets, 34(2), pp. 192–198. [CrossRef]
Chang, I.-S. , 1996, “ Investigation of Space Launch Vehicle Catastrophic Failures,” J. Spacecr. Rockets, 33(2), pp. 198–205. [CrossRef]
Kapurch, S. J. , 2007, NASA Systems Engineering Handbook, The National Aeronautics and Space Administration, Diane Publishing, Collingsdale, PA, p. 360.
Cardin, M.-A. , Nuttall, W. J. , De Neufville, R. , and Dahlgren, J. , 2007, “ Extracting Value From Uncertainty: A Methodology for Engineering Systems Design,” 17th Symposium International Council on System Engineering, San Diego, CA, June 24–28, pp. 1–9.
Forsberg, K. , and Mooz, H. , 1994, “ The Relationship of System Engineering to the Project Cycle,” 12th INTERNET World Congress Project Management, Oslo, Norway, June 9–11, p. 12.
Keating, C. , Rogers, R. , Unal, R. , Dryer, D. , Sousa-Poza, A. , Safford, R. , Peterson, W. , and Rabadi, G. , 2008, System of Systems Engineering, IEEE Eng. Manage. Rev., 36(4), p. 62.
Clark, J. O. , 2009, “ System of Systems Engineering and Family of Systems Engineering From a Standards, V-Model, and Dual-V Model Perspective,” Third Annual IEEE System Conference, Vancouver, BC, Canada, Mar. 23–26, pp. 381–387.
Collopy, P. D. , and Hollingsworth, P. M. , 2011, “ Value-Driven Design,” J. Aircr., 48(3), pp. 749–759. [CrossRef]
Collopy, P. D. , Bloebaum, C. L. , and Mesmer, B. L. , 2012, “ The Distinct and Interrelated Roles of Value-Driven Design, Multidisciplinary Design Optimization, and Decision Analysis,” AIAA Paper No. 2012-5575.
Mullan, C. , Price, M. , Soban, D. , and Fanthorpe, C. , 2012, “ Development of Subsystem Relationships Using a Value Driven Design Framework,” AIAA Paper No. 2012-5459.
Bertoni, M. , Bertoni, A. , and Isaksson, O. , 2016, “ Evoke: A Value-Driven Concept Selection Method for Early System Design,” J. Syst. Sci. Syst. Eng., 27(1), pp. 46–77. [CrossRef]
Kahneman, D. , Slovic, P. , and Tversky, A. , 1982, Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press, Cambridge, UK.
Slovic, P. , Finucane, M. , Peters, E. , and MacGregor, D. G. , 2002, “ Rational Actors or Rational Fools: Implications of the Affect Heuristic for Behavioral Economics,” J. Socio. Econ., 31(4), pp. 329–342. [CrossRef]
Wertz, J. R. , Everett, D. F. , and Puschell, J. J. , 2011, Space Mission Engineering: the New SMAD, Microcosm Press, Portland, OR.
Collopy, P. D. , 2012, “ A Research Agenda for the Coming Renaissance,” AIAA Paper No. 2012-799.
Kim, H. M. , Michelena, N. F. , Papalambros, P. Y. , and Jiang, T. , 2003, “ Target Cascading in Optimal System Design,” ASME J. Mech. Des., 125(3), p. 474. [CrossRef]
Tibor, E. B. , Mesmer, B. L. , Bloebaum, C. L. , Miller, S. W. , and Simpson, T. W. , 2014, “ Visualization of System Decomposition in a Value-Based Framework,” AIAA Paper No. 2014-2181.
Kannan, H. , Tibor, E. , Mesmer, B. , and Bloebaum, C. L. , 2015, “ Incorporation of Coupling Strength Models in a Value-Based Systems Engineering Framework for Optimization,” AIAA Paper No. 2015-3086.
Kwasa, B. , Bloebaum, C. L. , and Mesmer, B. , 2015, “ Value Impact of an Organization Structure in the Context of Value-Driven Design,” AIAA Paper No. 2015-1364.
Bhatia, G. V. , Kannan, H. , and Bloebaum, C. L. , 2016, “ A Game Theory Approach to Bargaining Over Attributes of Complex Systems in the Context of Value-Driven Design,” AIAA Paper No. 2016-0972.
Kannan, H. , Mesmer, B. , and Bloebaum, C. L. , 2016, “ Incorporation of Risk Preferences in a Value-Based Systems Engineering Framework for a Satellite System,” AIAA Paper No. 2016-0681.
Subramanian, T. R. , Khol, A. R. , Kannan, H. , Winer, E. H. , Bloebaum, C. L. , and Mesmer, B. , 2016, “ Understanding the Impact of Uncertainty on the Fidelity of the Value Model,” AIAA Paper No. 2016-0973.
Herrmann, J. W. , 2015, Engineering Decision Making and Risk Management, John Wiley & Sons, Hoboken, NJ.
Austin-Breneman, J. , Yu, B. Y. , and Yang, M. C. , 2015, “ Biased Information Passing Between Subsystems Over Time in Complex System Design,” ASME J. Mech. Des., 138(1), p. 011101. [CrossRef]
Takamatsu, T. , Hashimoto, I. , and Ohno, H. , 1970, “ Optimal Design of a Large Complex System From the Viewpoint of Sensitivity Analysis,” Ind. Eng. Chem. Process Des. Dev., 9(3), pp. 368–379. [CrossRef]
Thunnisen, D. P. , 2004, “ Method for Determining Margins in Conceptual Design,” J. Spacecr. Rockets, 41(1), pp. 85–92. [CrossRef]
Gu, X. , Renaud, J. , Batill, S. , Brach, R. , and Budhiraja, A. , 2000, “ Worst Case Propagated Uncertainty of Multidisciplinary Systems in Robust Design Optimization,” Struct. Multidiscip. Optim., 20(3), pp. 190–213. [CrossRef]
Helton, J. C. , 2011, “ Quantification of Margins and Uncertainties: Conceptual and Computational Basis,” Reliab. Eng. Syst. Saf., 96(9), pp. 976–1013. [CrossRef]
Sentz, K. , and Ferson, S. , 2011, “ Probabilistic Bounding Analysis in the Quantification of Margins and Uncertainties,” Reliab. Eng. Syst. Saf., 96(9), pp. 1126–1136. [CrossRef]
Anderson, N. H. , and Barrios, A. A. , 1961, “ Primacy Effects in Personality Impression Formation,” J. Abnorm. Soc. Psychol., 63(2), pp. 346–350. [CrossRef] [PubMed]
Wallsten, T. S. , 1981, “ Physician and Medical Student Bias in Evaluating Diagnostic Information,” Med. Decis. Making, 1(2), pp. 145–64. [CrossRef] [PubMed]
Wallsten, T. S. , 1983, “ The Theoretical Status of Judgmental Heuristics,” Adv. Psychol., 16, pp. 21–37. [CrossRef]
Slovic, P. , and Lichtenstein, S. , 1971, “ Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment,” Organ. Behav. Hum. Perform., 6(6), pp. 649–744. [CrossRef]
Finucane, M. L. , Alhakami, A. , Slovic, P. , and Johnson, S. M. , 2000, “ The Affect Heuristic in Judgments of Risks and Benefits,” J. Behav. Decis. Mak., 13(1), pp. 1–17. [CrossRef]
Macy, M. W. , and Willer, R. , 2002, “ From Factors to Factors: Computational Sociology and Agent-Based Modeling,” Annu. Rev. Sociol., 28(1), pp. 143–166. [CrossRef]
Macal, C. M. , and North, M. J ., 2008, “ Agent-Based Modeling and Simulation,” Proceedings of the 40th Conference on Winter Simulation, Miami, FL, Dec. 7–10, pp. 101–112.
McComb, C. , Cagan, J. , and Kotovsky, K. , 2016, “ Drawing Inspiration From Human Design Teams For Better Search And Optimization: The Heterogeneous Simulated Annealing Teams Algorithm,” ASME J. Mech. Des., 138(4), p. 044501. [CrossRef]
Triki, E. , Collette, Y. , and Siarry, P. , 2005, “ A Theoretical Study on the Behavior of Simulated Annealing Leading to a New Cooling Schedule,” Eur. J. Oper. Res., 166(1), pp. 77–92. [CrossRef]
Patton, M. Q. , 2002, Qualitative Research and Evaluation Methods, SAGE, Thousand Oaks, CA.
Anderson, J. C. , and Rungtusanatham, M. , 1994, “ A Theory of Quality Management Underlying the Deming Management Method,” Acad. Manag. Rev., 19(3), pp. 472–509. [CrossRef]
Azzalini, A. , 1985, “ A Class of Distributions Which Includes the Normal Ones,” Scand. J. Stat., 12(2), pp. 171–178. https://www.jstor.org/stable/4615982
Glaser, B. G. , and Strauss, A. L. , 1967, The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine Transactions, New Brunswick, NJ.
Corbin, J. M. , and Strauss, A. , 2008, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, SAGE, Thousand Oaks, CA.


Grahic Jump Location
Fig. 1

The information flowchart for the agent-based model. Arrows indicate the direction and content of information exchanged between the agents during each design cycle.

Grahic Jump Location
Fig. 2

Graphs showing the mass estimates for each agent and their respective utilities. Note the differences in the number of estimates and the skew of the utility functions for the subsystems and system.

Grahic Jump Location
Fig. 3

Each agent's uncertainty as a function of number of design cycles. Subsystem 2 does not converge to the prescribed uncertainty threshold due to bias.

Grahic Jump Location
Fig. 4

Histograms of the 1000 trials for each of the unbiased and biased cases, respectively

Grahic Jump Location
Fig. 5

Plots of the mean and variance for the system performance δξ as a function of the subsystem skew parameter αi and system skew parameter αξ



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In