Research Papers: Design Theory and Methodology

Ranking Ideas for Diversity and Quality

[+] Author and Article Information
Faez Ahmed

Department of Mechanical Engineering,
University of Maryland,
College Park, MD 20742
e-mail: faez00@umd.edu

Mark Fuge

Department of Mechanical Engineering,
University of Maryland,
College Park, MD 20742
e-mail: fuge@umd.edu

Contributed by the Design Theory and Methodology Committee of ASME for publication in the JOURNAL OF MECHANICAL DESIGN. Manuscript received December 15, 2016; final manuscript received September 21, 2017; published online November 9, 2017. Assoc. Editor: Katja Holtta-Otto.

J. Mech. Des 140(1), 011101 (Nov 09, 2017) (11 pages) Paper No: MD-16-1826; doi: 10.1115/1.4038070 History: Received December 15, 2016; Revised September 21, 2017

When selecting ideas or trying to find inspiration, designers often must sift through hundreds or thousands of ideas. This paper provides an algorithm to rank design ideas such that the ranked list simultaneously maximizes the quality and diversity of recommended designs. To do so, we first define and compare two diversity measures using determinantal point processes (DPP) and additive submodular functions. We show that DPPs are more suitable for items expressed as text and that a greedy algorithm diversifies rankings with both theoretical guarantees and empirical performance on what is otherwise an NP-Hard problem. To produce such rankings, this paper contributes a novel way to extend quality and diversity metrics from sets to permutations of ranked lists. These rank metrics open up the use of multi-objective optimization to describe trade-offs between diversity and quality in ranked lists. We use such trade-off fronts to help designers select rankings using indifference curves. However, we also show that rankings on trade-off front share a number of top-ranked items; this means reviewing items (for a given depth like the top ten) from across the entire diversity-to-quality front incurs only a marginal increase in the number of designs considered. While the proposed techniques are general purpose enough to be used across domains, we demonstrate concrete performance on selecting items in an online design community (OpenIDEO), where our approach reduces the time required to review diverse, high-quality ideas from around 25 h to 90 min. This makes evaluation of crowd-generated ideas tractable for a single designer. Our code is publicly accessible for further research.

Copyright © 2018 by ASME
Your Session has timed out. Please sign back in to continue.


Pauling, L. , 2001, Linus Pauling: Selected Scientific Papers, Vol. 2, World Scientific, Singapore.
Ahmed, F. , Fuge, M. , and Gorbunov, L. D. , 2016, “ Discovering Diverse, High Quality Design Ideas From a Large Corpus,” ASME Paper No. DETC2016-59926.
Shah, J. J. , Kulkarni, S. V. , and Vargas-Hernandez, N. , 2000, “ Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments,” ASME J. Mech. Des., 122(4), pp. 377–384. [CrossRef]
Verhaegen, P.-A. , Vandevenne, D. , Peeters, J. , and Duflou, J. R. , 2013, “ Refinements to the Variety Metric for Idea Evaluation,” Des. Stud., 34(2), pp. 243–263. [CrossRef]
Hennessey, B. A. , and Amabile, T. M. , 1999, “ Consensual Assessment,” Encycl. Creativity, 1, pp. 347–359.
Fuge, M. , Stroud, J. , and Agogino, A. , 2013, “ Automatically Inferring Metrics for Design Creativity,” ASME Paper No. DETC2013-12620.
Kudrowitz, B. M. , and Wallace, D. , 2013, “ Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation,” J. Eng. Des., 24(2), pp. 120–139. [CrossRef]
Green, M. , Seepersad, C. C. , and Hölttä-Otto, K. , 2014, “ Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study,” ASME Paper No. DETC2014-34434.
Von Hippel, E. , 2005, “ Democratizing Innovation: The Evolving Phenomenon of User Innovation,” J. Für Betriebswirtschaft, 55(1), pp. 63–78. [CrossRef]
Chiu, I. , and Shu, L. , 2012, “ Investigating Effects of Oppositely Related Semantic Stimuli on Design Concept Creativity,” J. Eng. Des., 23(4), pp. 271–296. [CrossRef]
Ali, K. , and Van Stam, W. , 2004, “ Tivo: Making Show Recommendations Using a Distributed Collaborative Filtering Architecture,” Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Seattle, WA, Aug. 22–25, pp. 394–401. http://citeseerx.ist.psu.edu/viewdoc/download?doi=
Coombs, C. H. , and Avrunin, G. S. , 1977, “ Single-Peaked Functions and the Theory of Preference,” Psychol. Rev., 84(2), p. 216. [CrossRef]
Ziegler, C.-N. , McNee, S. M. , Konstan, J. A. , and Lausen, G. , 2005, “ Improving Recommendation Lists Through Topic Diversification,” 14th International Conference on World Wide Web (WWW), Chiba, Japan, May 10–14, pp. 22–32. http://citeseerx.ist.psu.edu/viewdoc/download?doi=
Puthiya Parambath, S. A. , Usunier, N. , and Grandvalet, Y. , 2016, “ A Coverage-Based Approach to Recommendation Diversity on Similarity Graph,” Tenth ACM Conference on Recommender Systems (RecSys), Boston, MA, Sept. 15–19, pp. 15–22.
Santos, R. L. , Macdonald, C. , and Ounis, I. , 2010, “ Exploiting Query Reformulations for Web Search Result Diversification,” 19th International Conference on World Wide Web (WWW), Raleigh, NC, Apr. 26–30, pp. 881–890. http://wwwconference.org/proceedings/www2010/www/p881.pdf
Zhang, B. , Li, H. , Liu, Y. , Ji, L. , Xi, W. , Fan, W. , Chen, Z. , and Ma, W.-Y. , 2005, “ Improving Web Search Results Using Affinity Graph,” 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Salvador, Brazil, Aug. 15–19, pp. 504–511.
He, J. , Tong, H. , Mei, Q. , and Szymanski, B. , 2012, “ Gender: A Generic Diversified Ranking Algorithm,” Advances in Neural Information Processing Systems (NIPS), Stateline, NV, Dec. 3–8, pp. 1151–1159. https://papers.nips.cc/paper/4647-gender-a-generic-diversified-ranking-algorithm.pdf
Vargas, S. , and Castells, P. , 2011, “ Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems,” Fifth ACM Conference on Recommender Systems (RecSys), Chicago, IL, Oct. 23–27, pp. 109–116.
Castells, P. , Hurley, N. J. , and Vargas, S. , 2015, “ Novelty and Diversity in Recommender Systems,” Recommender Systems Handbook, Springer, New York, pp. 881–918. [CrossRef]
Zhang, Y. C. , Séaghdha, D. Ó. , Quercia, D. , and Jambor, T. , 2012, “ Auralist: Introducing Serendipity Into Music Recommendation,” Fifth ACM International Conference on Web Search and Data Mining (WSDM), Seattle, WA, Feb. 8–12, pp. 13–22.
Fisher, D. , Jain, A. , Keikha, M. , Croft, W. , and Lipka, N. , 2015, “ Evaluating Ranking Diversity and Summarization in Microblogs Using Hashtags,” University of Massachusetts, Boston, MA, Technical Report. https://pdfs.semanticscholar.org/9f0c/53afcc5e33b8b22722add0812bf14ccf875b.pdf
Patil, G. , and Taillie, C. , 1982, “ Diversity as a Concept and Its Measurement,” J. Am. Stat. Assoc., 77(379), pp. 548–561. [CrossRef]
Zhu, X. , Goldberg, A. B. , Van Gael, J. , and Andrzejewski, D. , 2007, “ Improving Diversity in Ranking Using Absorbing Random Walks,” Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Rochester, NY, Apr. 22–27, pp. 97–104. http://pages.cs.wisc.edu/~jerryzhu/pub/grasshopper.pdf
Zhao, P. , and Lee, D. L. , 2016, “ How Much Novelty Is Relevant? It Depends on Your Curiosity,” 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Pisa, Italy, July 17–21, pp. 315–324.
Wang, X. , Dou, Z. , Sakai, T. , and Wen, J.-R. , 2016, “ Evaluating Search Result Diversity Using Intent Hierarchies,” 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Pisa, Italy, July 17–22, pp. 415–424.
Chapelle, O. , Ji, S. , Liao, C. , Velipasaoglu, E. , Lai, L. , and Wu, S.-L. , 2011, “ Intent-Based Diversification of Web Search Results: Metrics and Algorithms,” Inf. Retr., 14(6), pp. 572–592. [CrossRef]
Clarke, C. L. , Kolla, M. , Cormack, G. V. , Vechtomova, O. , Ashkan, A. , Büttcher, S. , and MacKinnon, I. , 2008, “ Novelty and Diversity in Information Retrieval Evaluation,” 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Singapore, July 20–24, pp. 659–666.
Carterette, B. , 2009, “ An Analysis of Np-Completeness in Novelty and Diversity Ranking,” International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory (ICTIR), Cambridge, UK, Sept. 10–12, pp. 200–211.
Chen, W. , Chazan, J. , and Fuge, M. , 2016, “ How Designs Differ: Non-Linear Embeddings Illuminate Intrinsic Design Complexity,” ASME Paper No. DETC2016-60112.
Yumer, M. E. , Asente, P. , Mech, R. , and Kara, L. B. , 2015, “ Procedural Modeling Using Autoencoder Networks,” 28th Annual ACM Symposium on User Interface Software & Technology (UIST), Charlotte, NC, Nov. 11–15, pp. 109–118.
Burnap, A. , Pan, Y. , Liu, Y. , Ren, Y. , Lee, H. , Gonzalez, R. , and Papalambros, P. Y. , 2016, “ Improving Design Preference Prediction Accuracy Using Feature Learning,” ASME J. Mech. Des., 138(7), p. 071404. [CrossRef]
Yu, Q. , Yang, Y. , Song, Y.-Z. , Xiang, T. , and Hospedales, T. , 2015, “ Sketch-A-Net That Beats Humans,” Int. J. Com. Vision, 122(3), pp. 411–425 http://www.eecs.qmul.ac.uk/~tmh/papers/yu2015sketchanet.pdf.
Dong, A. , 2005, “ The Latent Semantic Approach to Studying Design Team Communication,” Des. Stud., 26(5), pp. 445–461. [CrossRef]
Pu, Y. , Gan, Z. , Henao, R. , Yuan, X. , Li, C. , Stevens, A. , and Carin, L. , 2016, “ Variational Autoencoder for Deep Learning of Images, Labels and Captions,” Advances in Neural Information Processing Systems (NIPS), Barcelona, Spain, Dec. 5–10, pp. 2352–2360. https://zhegan27.github.io/Papers/vae_nips2016_poster.pdf
Tamuz, O. , Liu, C. , Belongie, S. , Shamir, O. , and Kalai, A. T. , 2011, “ Adaptively Learning the Crowd Kernel,” International Conference on Machine Learning (ICML), Bellevue, WA, June 28–July 2, pp. 673–680 https://dl.acm.org/citation.cfm?id=3104567.
Qian, L. , and Gero, J. S. , 1996, “ Function–Behavior–Structure Paths and Their Role in Analogy-Based Design,” Artificial Intell. Eng., Des., Anal. Manuf., 10(4), pp. 289–312. [CrossRef]
Kirschman, C. , Fadel, G. , and Jara-Almonte, C. , 1998, “ Classifying Functions for Mechanical Design,” ASME J. Mech. Des., 120(3), pp. 475–482. [CrossRef]
Stone, R. B. , and Wood, K. L. , 2000, “ Development of a Functional Basis for Design,” ASME J. Mech. Des., 122(4), pp. 359–370. [CrossRef]
Vishwanathan, S. V. N. , Schraudolph, N. N. , Kondor, R. , and Borgwardt, K. M. , 2010, “ Graph Kernels,” J. Mach. Learn. Res., 11, pp. 1201–1242. http://www.jmlr.org/papers/v11/vishwanathan10a.html
Lin, H. , and Bilmes, J. , 2011, “ A Class of Submodular Functions for Document Summarization,” 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT), Portland, OR, June 19–24, pp. 510–520. https://dl.acm.org/citation.cfm?id=2002537
Lin, H. , and Bilmes, J. A. , 2012, “ Learning Mixtures of Submodular Shells With Application to Document Summarization,” Twenty-Eighth Conference on Uncertainty in Artificial Intelligence (UAI), Catalina Island, CA, Aug. 14–18, pp. 479–490 https://dl.acm.org/citation.cfm?id=3020652.3020704.
Kulesza, A. , and Taskar, B. , 2012, Determinantal Point Processes for Machine Learning, Now Publishers Inc., Hanover, MA.
Boim, R. , Milo, T. , and Novgorodov, S. , 2011, “ Diversification and Refinement in Collaborative Filtering Recommender,” 20th ACM International Conference on Information and Knowledge Management (CIKM), Glasgow, Scotland, Oct. 24–28, pp. 739–744.
Feige, U. , Mirrokni, V. S. , and Vondrak, J. , 2011, “ Maximizing Non-Monotone Submodular Functions,” SIAM J. Comput., 40(4), pp. 1133–1153. [CrossRef]
Manning, C. D. , and Schütze, H. , 1999, Foundations of Statistical Natural Language Processing, Vol. 999, MIT Press, Cambridge, MA.
Ng, A. Y. , Jordan, M. I. , and Weiss, Y. , 2002, “ On Spectral Clustering: Analysis and An Algorithm,” Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada, Dec. 3–8, pp. 849–856. https://dl.acm.org/citation.cfm?id=2980649
Kulesza, A. , and Taskar, B. , 2011, “ Learning Determinantal Point Processes,” 27th Conference on Uncertainty in Artificial Intelligence (UAI), Barcelona, Spain, July 14–17, pp. 1–9. https://homes.cs.washington.edu/~taskar/pubs/ldpps_uai11.pdf
Kulesza, A. , and Taskar, B. , 2011, “ k-Dpps: Fixed-Size Determinantal Point Processes,” 28th International Conference on Machine Learning (ICML), Bellevue, WA, June 28–July 2, pp. 1193–1200. https://homes.cs.washington.edu/~taskar/pubs/kdpps_icml11.pdf
Borodin, A. , 2009, “ Determinantal Point Processes,” preprint arXiv:0911.1153. https://arxiv.org/abs/0911.1153
Toubia, O. , and Florès, L. , 2007, “ Adaptive Idea Screening Using Consumers,” Mark. Sci., 26(3), pp. 342–360. [CrossRef]
Mollick, E. , and Nanda, R. , 2015, “ Wisdom or Madness? Comparing Crowds With Expert Evaluation in Funding the Arts,” Manage. Sci., 62(6), pp. 1533–1553. [CrossRef]
Ahmed, F. , and Fuge, M. , 2017, “ Capturing Winning Ideas in Online Design Communities,” 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW), Portland, OR, Feb. 25–Mar. 1, pp. 1675–1687.
Järvelin, K. , and Kekäläinen, J. , 2002, “ Cumulated Gain-Based Evaluation of IR Techniques,” ACM Trans. Inf. Syst., 20(4), pp. 422–446.
Carbonell, J. , and Goldstein, J. , 1998, “ The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries,” 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), Melbourne, Australia, Aug. 24–28, pp. 335–336.
Deb, K. , Pratap, A. , Agarwal, S. , and Meyarivan, T. , 2002, “ A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II,” IEEE Trans. Evol. Comput., 6(2), pp. 182–197. [CrossRef]
Fuge, M. , Tee, K. , Agogino, A. , and Maton, N. , 2014, “ Analysis of Collaborative Design Networks: A Case Study of OpenIDEO,” ASME J. Comput. Inf. Sci. Eng., 14(2), p. 021009. [CrossRef]
Chiu, P.-W. , and Bloebaum, C. , 2008, “ Hyper-Radial Visualization (HRV) With Weighted Preferences for Multi-Objective Decision Making,” AIAA Paper No. 2008-5986.
Hofmann, K. , Whiteson, S. , and Rijke, M. D. , 2013, “ Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods,” ACM Trans. Inf. Syst. (TOIS), 31(4), p. 17. [CrossRef]
Deb, K. , and Gupta, S. , 2011, “ Understanding Knee Points in Bicriteria Problems and Their Implications as Preferred Solution Principles,” Eng. Optim., 43(11), pp. 1175–1204. [CrossRef]
Jain, L. , Jamieson, K. G. , and Nowak, R. , 2016, “ Finite Sample Prediction and Recovery Bounds for Ordinal Embedding,” Advances in Neural Information Processing Systems (NIPS), Barcelona, Spain, Dec. 5–10, pp. 2703–2711. https://papers.nips.cc/paper/6554-finite-sample-prediction-and-recovery-bounds-for-ordinal-embedding
Chakrabarti, A. , Shea, K. , Stone, R. , Cagan, J. , Campbell, M. , Hernandez, N. V. , and Wood, K. L. , 2011, “ Computer-Based Design Synthesis Research: An Overview,” ASME J. Comput. Inf. Sci. Eng., 11(2), p. 021003. [CrossRef]


Grahic Jump Location
Fig. 1

Trade-off front between diversity and quality of ranked lists. Each point is a different permutation of 606 ideas. A is the most diverse solution while C is the solution with highest quality objective. Indifference curves are used to find the point B closest to the ideal point.

Grahic Jump Location
Fig. 2

Ideas selected in top ten of different solution sets on the trade-off front between quality and diversity. The figure shows that only a small set of 36 unique ideas appear on trade-off front (the lines in the figures). On the bottom are ideas selected for high quality in the trade-off front, while top of the figure has ideas with high diversity.

Grahic Jump Location
Fig. 3

Determinant of subsets for different ranked lists. The 5th and 95th percentile solutions show that marginal gain in diversity after 60 solutions is very low. The most diverse solution (A) from trade-off front selected using greedy solution is significantly more diverse than random permutations.



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In