Research Papers: Design Theory and Methodology

Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel

[+] Author and Article Information
Faez Ahmed

Department of Mechanical Engineering,
University of Maryland,
College Park, MD 20742
e-mail: faez00@umd.edu

Sharath Kumar Ramachandran

School of Engineering Design,
Technology and Professional Programs,
The Pennsylvania State University,
University Park, PA 16801
e-mail: sharath@psu.edu

Mark Fuge

Department of Mechanical Engineering,
University of Maryland,
College Park, MD 20742
e-mail: fuge@umd.edu

Samuel Hunter

Industrial and Organizational Psychology,
The Pennsylvania State University,
University Park, PA 16801
e-mail: sth11@psu.edu

Scarlett Miller

School of Engineering Design,
Technology and Professional Programs,
The Pennsylvania State University,
University Park, PA 16801
e-mail: shm13@psu.edu

1Corresponding author.

Open-source code implementing our approach is available at: https://github.com/IDEALLAB/idea_mapContributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received June 30, 2018; final manuscript received October 19, 2018; published online December 20, 2018. Assoc. Editor: Katja Holtta-Otto.

J. Mech. Des 141(2), 021102 (Dec 20, 2018) (13 pages) Paper No: MD-18-1514; doi: 10.1115/1.4041856 History: Received June 30, 2018; Revised October 19, 2018

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets to score ideas on their uniqueness. However, their decisions about novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two-dimensional (2D) idea maps. We derive these maps by asking participants for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the design domain. We also propose that novel ideas can be identified by finding outliers on these idea maps. To demonstrate our method, we conduct experimental evaluations on two datasets—colored polygons (known answer) and milk frother sketches (unknown answer). We show that idea maps shed light on factors considered by participants in judging idea similarity and the maps are robust to noisy ratings. We also compare physical maps made by participants on a white-board to their computationally generated idea maps to compare how people think about spatial arrangement of design items. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

Copyright © 2019 by ASME
Topics: Design , Errors
Your Session has timed out. Please sign back in to continue.


Starkey, E. , Toh, C. A. , and Miller, S. R. , 2016, “ Abandoning Creativity: The Evolution of Creative Ideas in Engineering Design Course Projects,” Des. Stud., 47, pp. 47–72. [CrossRef]
Hammedi, W. , van Riel, A. C. , and Sasovova, Z. , 2011, “ Antecedents and Consequences of Reflexivity in New Product Idea Screening,” J. Prod. Innovation Manage., 28(5), pp. 662–679.
Lopez-Mesa, B. , and Vidal, R. , 2006, “ Novelty Metrics in Engineering Design Experiments,” Ninth International Design Conference (DESIGN), Dubrovnik, Croatia, May 15–18, pp. 557–564.
Sarkar, P. , and Chakrabarti, A. , 2011, “ Assessing Design Creativity,” Des. Stud., 32(4), pp. 348–383. [CrossRef]
Johnson, T. A. , Cheeley, A. , Caldwell, B. W. , and Green, M. G. , 2016, “ Comparison and Extension of Novelty Metrics for Problem-Solving Tasks,” ASME Paper No. DETC2016-60319.
Simonton, D. K. , 2012, “ Taking the U.S. Patent Office Criteria Seriously: A Quantitative Three-Criterion Creativity Definition and Its Implications,” Creativity Res. J., 24(2–3), pp. 97–106. [CrossRef]
Maher, M. L. , and Fisher, D. H. , 2012, “ Using AI to Evaluate Creative Designs,” Second International Conference on Design Creativity, Glasgow, UK, Sept. 18–20, pp. 45–54. https://www.designsociety.org/download-publication/32461/Using+AI+to+Evaluate+Creative+Designs
Verhaegen, P.-A. , Vandevenne, D. , and Duflou, J. , 2012, “ Originality and Novelty: A Different Universe,” 12th International Design Conference, Dubrovnik, Croatia, May 21–24, pp. 1961–1966.
Seymore, S. B. , 2011, “ Rethinking Novelty in Patent Law,” Duke Law J., 60, pp. 919–976. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1558617
Chen, L. , Xu, P. , and Liu, D. , 2015, “ Experts Versus the Crowd: A Comparison of Selection Mechanisms in Crowdsourcing Contests,” SSRN Electronic J. (epub).
Chen, L. , and Liu, D. , 2012, “ Comparing Strategies Winning Expert-Rated Crowd-Rated Crowdsourcing Contests: First Findings,” 18th Americas Conference on Information Systems (AMCIS), Seattle, WA, Aug. 9–12, pp. 97–107.
Green, M. , Seepersad, C. C. , and Hölttä-Otto, K. , 2014, “ Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study,” ASME Paper No. .
Surowiecki, J. , 2004, The wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations, Vol. 296, American Psychological Association, Washington, DC.
Yu, B. , Willis, M. , Sun, P. , and Wang, J. , 2013, “ Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk,” J. Med. Internet Res., 15(6), p. e108. https://www.jmir.org/2013/6/e108/
Wu, H. , Corney, J. , and Grant, M. , 2015, “ An Evaluation Methodology for Crowdsourced Design,” Adv. Eng. Inf., 29(4), pp. 775–786. [CrossRef]
Görzen, T. , and Kundisch, D. , 2016, “ Can the Crowd Substitute Experts in Evaluating Creative Jobs? the Case of Business Models,” European Conference on Information Systems, Rome, Italy, Sept. 4–9.
Hennessey, B. A. , and Amabile, T. M. , 1999, “ Consensual Assessment,” Encycl. Creativity, 1, pp. 347–359.
Licuanan, B. F. , Dailey, L. R. , and Mumford, M. D. , 2007, “ Idea Evaluation: Error in Evaluating Highly Original Ideas,” J. Creative Behav., 41(1), pp. 1–27. [CrossRef]
Shah, J. J. , Kulkarni, S. V. , and Vargas-Hernandez, N. , 2000, “ Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments,” ASME J. Mech. Des., 122(4), pp. 377–384. [CrossRef]
Verhaegen, P.-A. , Vandevenne, D. , Peeters, J. , and Duflou, J. R. , 2013, “ Refinements to the Variety Metric for Idea Evaluation,” Des. Stud., 34(2), pp. 243–263. [CrossRef]
Shah, J. J. , Smith, S. M. , and Vargas-Hernandez, N. , 2003, “ Metrics for Measuring Ideation Effectiveness,” Des. Stud., 24(2), pp. 111–134. [CrossRef]
Oman, S. K. , Tumer, I. Y. , Wood, K. , and Seepersad, C. , 2013, “ A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects,” Res. Eng. Des., 24(1), pp. 65–92. [CrossRef]
Brown, D. C. , 2014, “ Problems With the Calculation of Novelty Metrics,” Sixth International Conference on Design Computing and Cognition (DCC'14), London, June 23–25. https://pdfs.semanticscholar.org/449d/1bbdadd385aa6055b3f1ef1c10804169b53a.pdf
Baer, J. , 2012, “ Domain Specificity and the Limits of Creativity Theory,” J. Creative Behav., 46(1), pp. 16–29. [CrossRef]
Casakin, H. , and Kreitler, S. , 2005, “ The Nature of Creativity in Design,” Studying Designers, 5, pp. 87–100.
Richardson, T. , Nekolny, B. , Holub, J. , and Winer, E. H. , 2014, “ Visualizing Design Spaces Using Two-Dimensional Contextual Self-Organizing Maps,” AIAA J., 52(4), pp. 725–738. [CrossRef]
Tang, J. , Liu, J. , Zhang, M. , and Mei, Q. , 2016, “ Visualizing Large-Scale and High-Dimensional Data,” 25th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee, Montreal, QC, Canada, Apr. 11–15, pp. 287–297.
Maaten, L. V. D. , and Hinton, G. , 2008, “ Visualizing Data Using t-Sne,” J. Mach. Learn. Res., 9, pp. 2579–2605. http://www.jmlr.org/papers/v9/vandermaaten08a.html
Chen, W. , Fuge, M. , and Chazan, N. , 2017, “ Design Manifolds Capture the Intrinsic Complexity and Dimension of Design Spaces,” ASME J. Mech. Des., 139(5), p. 051102. [CrossRef]
Li, L. , Malave, V. , Song, A. , and Yu, A. J. , 2016, “ Extracting Human Face Similarity Judgments: Pairs or Triplets?,” J. Vision, 16(12), pp. 719–719. [CrossRef]
Torgerson, W. S. , 1958, Theory and Methods of Scaling, Wiley, Hoboken, NJ.
van der Maaten, L. , and Weinberger, K. , 2012, “ Stochastic Triplet Embedding,” IEEE International Workshop on Machine Learning for Signal Processing, Santander, Spain, Sept. 23–26, pp. 1–6.
Stewart, N. , Brown, G. D. , and Chater, N. , 2005, “ Absolute Identification by Relative Judgment,” Psychol. Rev., 112(4), p. 881. [CrossRef] [PubMed]
Agarwal, S. , Wills, J. , Cayton, L. , Lanckriet, G. , Kriegman, D. , and Belongie, S. , 2007, “ Generalized Non-Metric Multidimensional Scaling,” PMLR, 2, pp. 11–18.
Tamuz, O. , Liu, C. , Belongie, S. , Shamir, O. , and Kalai, A. T. , 2011, “ Adaptively Learning the Crowd Kernel,” 28th International Conference on Machine Learning (ICML'11), June 28–July 2, Bellevue, WA, pp. 673–680. https://people.csail.mit.edu/celiu/pdfs/2011_ICML_TamLiuBelShamKal.pdf
Sankaranarayanan, S. , Alavi, A. , and Chellappa, R. , 2016, “ Triplet Similarity Embedding for Face Verification,” Preprint arXiv: 1602.03418. https://arxiv.org/abs/1602.03418
Nhat, V. D. M. , Vo, D. , Challa, S. , and Lee, S. , 2008, “ Nonmetric MDS for Sensor Localization,” Third International Symposium on Wireless Pervasive Computing (ISWPC), Santorini, Greece, May 7–9, pp. 396–400.
Haghiri, S. , Ghoshdastidar, D. , and von Luxburg, U. , 2017, “ Comparison-Based Nearest Neighbor Search,” PMLR, 54, pp. 851–859. http://proceedings.mlr.press/v54/haghiri17a/haghiri17a.pdf
Ukkonen, A. , Derakhshan, B. , and Heikinheimo, H. , 2015. “ Crowdsourced Nonparametric Density Estimation Using Relative Distances,” Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP), San Diego, CA, Nov. 8–11, pp. 188–197. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP15/paper/viewFile/11586/11441
Demiralp, Ç. , Bernstein, M. S. , and Heer, J. , 2014, “ Learning Perceptual Kernels for Visualization Design,” IEEE Trans. Visualization Comput. Graphics, 20(12), pp. 1933–1942. [CrossRef]
Siangliulue, P. , Arnold, K. C. , Gajos, K. Z. , and Dow, S. P. , 2015, “ Toward Collaborative Ideation at Scale: Leveraging Ideas From Others to Generate More Creative and Diverse Ideas,” 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW), Vancouver, BC, Canada, Mar. 14–18, pp. 937–945. https://dl.acm.org/citation.cfm?id=2675239
Kawakita, J. , 1991, “ The Original KJ Method,” Tokyo: Kawakita Res. Inst., pp. 233–237.
Lin, H. , and Bilmes, J. , 2011, “ A Class of Submodular Functions for Document Summarization,” 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, June 19–24, pp. 510–520.
Gower, J. C. , 1975, “ Generalized Procrustes Analysis,” Psychometrika, 40(1), pp. 33–51. [CrossRef]
Toh, C. A. , and Miller, S. R. , 2016, “ Choosing Creativity: The Role of Individual Risk and Ambiguity Aversion on Creative Concept Selection in Engineering Design,” Res. Eng. Des., 27(3), pp. 195–219. [CrossRef]
Amid, E. , Vlassis, N. , and Warmuth, M. K. , 2016, “ Low-Dimensional Data Embedding Via Robust Ranking,” Preprint arXiv: 1611.09957. https://arxiv.org/abs/1611.09957
Hoffmann, H. , 2007, “ Kernel PCA for Novelty Detection,” Pattern Recognit., 40(3), pp. 863–874. [CrossRef]
Ahmed, F. , and Fuge, M. , 2018, “ Ranking Ideas for Diversity and Quality,” ASME J. Mech. Des., 140(1), p. 011101. [CrossRef]
Ahmed, F. , Fuge, M. , and Gorbunov, L. D. , 2016, “ Discovering Diverse, High Quality Design Ideas From a Large Corpus,” ASME Paper No. DETC2016-59926.
Wilber, M. , Kwak, I. S. , Kriegman, D. , and Belongie, S. , 2015, “ Learning Concept Embeddings With Combined Human-Machine Expertise,” International Conference on Computer Vision (ICCV), Santiago, Chile, Dec. 7–13, pp. 981–989. https://vision.cornell.edu/se3/wp-content/uploads/2015/09/main.pdf
Elsbach, K. D. , and Kramer, R. M. , 2003, “ Assessing Creativity in Hollywood Pitch Meetings: Evidence for a Dual-Process Model of Creativity Judgments,” Acad. Manage. J., 46(3), pp. 283–301.
Amid, E. , and Ukkonen, A. , 2015, “ Multiview Triplet Embedding: Learning Attributes in Multiple Maps,” International Conference on Machine Learning, Lille, France, July 6–11, pp. 1472–1480. https://pdfs.semanticscholar.org/0124/4108e4b6a190aa4c5085518e3c3412fa0087.pdf


Grahic Jump Location
Fig. 1

Example of triplet query asked from participants in our experiment. Participant answers the question: “Which design is more similar to design A?”.

Grahic Jump Location
Fig. 2

(a) Dataset of ten polygons used in first experiment and (b) two idea maps with four items each. Although these maps look different, they satisfy the same set of triplet queries.

Grahic Jump Location
Fig. 3

Two-dimensional embedding obtained for polygons using triplets given by the automated rater, who uses preset rules

Grahic Jump Location
Fig. 4

Two-dimensional embedding obtained from polygon dataset by participant 5, who uses number of sides as primary criteria the decisions

Grahic Jump Location
Fig. 5

Two-dimensional embedding obtained from polygon dataset by participant 9, who uses “color, shape, number of side” as criteria

Grahic Jump Location
Fig. 6

Ten milk frother sketches used in experimental 2

Grahic Jump Location
Fig. 7

Left: (a) Idea map of design sketches for participant 7. Center of the sketch represents the 2D position of embedding. Two main clusters can be seen. Right: (b) Idea map of design sketches for participant 10. Center of the sketch represents the 2D position of embedding.

Grahic Jump Location
Fig. 8

Idea map obtained by combining triplets from all participants. Id of each sketch is at bottom right corner.

Grahic Jump Location
Fig. 9

Left: (a) Triplet error between idea maps of embedding shown in Fig. 8 and embedding obtained using a subset of triplet ratings. 100 runs with different subsets used to obtain embeddings. Using only 30% of triplets, median error is less than 10%. Right: (b) Triplet error between embedding generated using noisy triplets compared to embedding shown in Fig. 8. We perform 100 runs and flip a subset of triplets randomly to obtain the embeddings. Small increase in the median error shows that idea maps are robust to small percentage of false ratings by participants.

Grahic Jump Location
Fig. 10

Left: (a) Participant creating a map by positioning idea sketches on provided canvas. Right: (b) correspondence between human generated map and triplet map for participant 10 after Procrustes transformation. Apart from sketch 4, most sketches have minor relative displacements.



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In