Abstract

The goal of this research is to develop a computer-aided visual analogy support (CAVAS) framework to augment designers’ visual analogical thinking by stimulating them by providing relevant visual cues from a variety of categories. Two steps are taken to reach this goal: developing a flexible computational framework to explore various visual cues, i.e., shapes or sketches, based on the relevant datasets and conducting human-based behavioral studies to validate such visual cue exploration tools. This article presents the results and insights obtained from the first step by addressing two research questions: How can the computational framework CAVAS be developed to provide designers in sketching with certain visual cues for stimulating their visual thinking process? How can a computation tool learn a latent space, which can capture the shape patterns of sketches? A visual cue exploration framework and a deep clustering model CAVAS-DL are proposed to learn a latent space of sketches that reveal shape patterns for multiple sketch categories and simultaneously cluster the sketches to preserve and provide category information as part of visual cues. The distance- and overlap-based similarities are introduced and analyzed to identify long- and short-distance analogies. Performance evaluations of our proposed methods are carried out with different configurations, and the visual presentations of the potential analogical cues are explored. The results have demonstrated the applicability of the CAVAS-DL model as the basis for the human-based validation studies in the next step.

References

1.
Jin
,
Y.
, and
Benami
,
O.
,
2010
, “
Creative Patterns and Stimulation in Conceptual Design
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
24
(
2
), pp.
191
209
.
2.
Goldschmidt
,
G.
,
2001
, “Visual Analogy—A Strategy for Design Reasoning andLearning,”
Design Knowing and Learning: Cognition in Design Education
,
C. M.
Eastman
,
W. M.
McCracken
, and
W. C.
Newstetter
, eds.,
Elsevier
,
New York
, pp.
199
219
.
3.
Mougenot
,
C.
,
Bouchard
,
C.
,
Aoussat
,
A.
, and
Westerman
,
S.
,
2008
, “
Inspiration, Images and Design: An Investigation of Designers’ Information Gathering Strategies
,”
J. Des. Res.
,
7
(
4
), pp.
331
351
.
4.
Bouchard
,
C.
,
Omhover
,
J.-f.
,
Mougenot
,
C.
,
Aoussat
,
A.
, and
Westerman
,
S. J.
,
2008
, “TRENDS: a Content-Based Information Retrieval System for Designers,”
Proc. Design Computing and Cognition ’08
, Vol.
4
,
Springer Science + Business Media B.V.
,
Atlanta, GA
, pp.
593
611
.
5.
Chakrabarti
,
A.
,
Siddharth
,
L.
,
Dinakar
,
M.
,
Panda
,
M.
,
Palegar
,
N.
, and
Keshwani
,
S.
,
2017
, “
Idea Inspire 3.0—A Tool for Analogical Design
,”
Proceedings International Conference on Research Into Design
,
Singapore
,
January
, Springer, pp.
475
485
.
6.
Han
,
J.
,
Shi
,
F.
,
Chen
,
L.
, and
Childs
,
P. R.
,
2018
, “
A Computational Tool for Creative Idea Generation Based on Analogical Reasoning and Ontology
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
32
(
4
), pp.
462
477
.
7.
Ullman
,
D. G.
,
Wood
,
S.
, and
Craig
,
D.
,
1990
, “
The Importance of Drawing in the Mechanical Design Process
,”
Comput. Graph.
,
14
(
2
), pp.
263
274
.
8.
Yang
,
M. C.
,
2009
, “
Observations on Concept Generation and Sketching in Engineering Design
,”
Res. Eng. Des.
,
20
(
1
), pp.
1
11
.
9.
Kokotovich
,
V.
, and
Purcell
,
T.
,
2000
, “
Mental Synthesis and Creativity in Design: An Experimental Examination
,”
Des. Stud.
,
21
(
5
), pp.
437
449
.
10.
Goldschmidt
,
G.
, and
Smolkov
,
M.
,
2006
, “
Variances in the Impact of Visual Stimuli on Design Problem Solving Performance
,”
Des. Stud.
,
27
(
5
), pp.
549
569
.
11.
Goldschmidt
,
G.
,
2003
, “
The Backtalk of Self-Generated Sketches
,”
Des. Issues
,
19
(
1
), pp.
72
88
.
12.
Casakin
,
H.
, and
Goldschmidt
,
G.
,
1999
, “
Expertise and the Use of Visual Analogy: Implications for Design Education
,”
Des. Stud.
,
20
(
2
), pp.
153
175
.
13.
Marshall
,
K. S.
,
Crawford
,
R.
, and
Jensen
,
D.
,
2016
, “
Analogy Seeded Mind-Maps: A Comparison of Verbal and Pictorial Representation of Analogies in the Concept Generation Process
,”
Proceedings ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
,
Aug. 21–24
, p. V007T006A010.
14.
Malaga
,
R. A.
,
2000
, “
The Effect of Stimulus Modes and Associative Distance in Individual Creativity Support Systems
,”
Decis. Support Syst.
,
29
(
2
), pp.
125
141
.
15.
Gero
,
J.
, and
Yan
,
M.
,
1994
, “
Shape Emergence by Symbolic Reasoning
,” Environ. Plann. B Plann. Des.,
21
(
2
), pp.
191
212
.
16.
Oxman
,
R.
,
2002
, “
The Thinking eye: Visual Re-Cognition in Design Emergence
,”
Des. Stud.
,
23
(
2
), pp.
135
164
.
17.
Arnheim
,
R.
,
1997
,
Visual Thinking
,
University of California Press
,
Berkeley, CA
.
18.
Chen
,
W.
,
Fuge
,
M.
, and
Chazan
,
J.
,
2017
, “
Design Manifolds Capture the Intrinsic Complexity and Dimension of Design Spaces
,”
ASME J. Mech. Des.
,
139
(
5
), p.
051102
.
19.
Chen
,
W.
, and
Fuge
,
M.
,
2017
, “
Beyond the Known: Detecting Novel Feasible Domains Over an Unbounded Design Space
,”
ASME J. Mech. Des.
,
139
(
11
), pp.
111405
111410
.
20.
Kwon
,
E.
,
Pehlken
,
A.
,
Thoben
,
K.-D.
,
Bazylak
,
A.
, and
Shu
,
L. H.
,
2019
, “
Visual Similarity to Aid Alternative-Use Concept Generation for Retired Wind-Turbine Blades
,”
ASME J. Mech. Des.
,
141
(
3
), p.
031106
.
21.
Casakin
,
H.
,
2010
, “
Visual Analogy, Visual Displays, and the Nature of Design Problems: The Effect of Expertise
,” Environ. Plann. B Plann. Des.,
37
(
1
), pp.
170
188
.
22.
Herring
,
S. R.
,
Chang
,
C.-C.
,
Krantzler
,
J.
, and
Bailey
,
B. P.
,
2009
, “
Getting Inspired!: Understanding How and Why Examples Are Used in Creative Design Practice
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Boston, MA
,
Apr. 4–9
, ACM, pp.
87
96
.
23.
Zhang
,
Z.
, and
Jin
,
Y.
,
2020
, “
An Unsupervised Deep Learning Model to Discover Visual Similarity Between Sketches for Visual Analogy Support
,”
Proceedings ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual
,
August
, p. V008T008A003.
24.
Linsey
,
J.
,
Wood
,
K. L.
, and
Markman
,
A. B.
,
2008
, “
Modality and Representation in Analogy
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
22
(
2
), pp.
85
100
.
25.
Vincent
,
J. F.
, and
Mann
,
D. L.
,
2002
, “
Systematic Technology Transfer From Biology to Engineering
,”
Philos. Trans. R. Soc. London, Ser. A
,
360
(
1791
), pp.
159
173
.
26.
Chakrabarti
,
A.
,
Sarkar
,
P.
,
Leelavathamma
,
B.
, and
Nataraju
,
B.
,
2005
, “
A Functional Representation for Aiding Biomimetic and Artificial Inspiration of New Ideas
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
19
(
2
), pp.
113
132
.
27.
Sarkar
,
P.
, and
Chakrabarti
,
A.
,
2008
, “
The Effect of Representation of Triggers on Design Outcomes
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
22
(
2
), pp.
101
116
.
28.
Cheong
,
H.
,
Chiu
,
I.
,
Shu
,
L.
,
Stone
,
R. B.
, and
McAdams
,
D. A.
,
2011
, “
Biologically Meaningful Keywords for Functional Terms of the Functional Basis
,”
ASME J. Mech. Des.
,
133
(
2
), p.
021007
.
29.
Shu
,
L.
,
2010
, “
A Natural-Language Approach to Biomimetic Design
,”
AI EDAM: Artificial Intelligence for Engineering Design, Analysis, and Manufacturing
,
24
, pp.
507
519
.
30.
Vattam
,
S.
,
Wiltgen
,
B.
,
Helms
,
M.
,
Goel
,
A. K.
, and
Yen
,
J.
,
2011
, “DANE: Fostering Creativity in and Through Biologically Inspired Design,”
Design Creativity 2010
,
Taura
T.
and
Nagai
Y.
ed.,
Springer
,
London
, pp.
115
122
.
31.
Goel
,
A. K.
,
Vattam
,
S.
,
Wiltgen
,
B.
, and
Helms
,
M.
,
2012
, “
Cognitive, Collaborative, Conceptual and Creative—Four Characteristics of the Next Generation of Knowledge-Based CAD Systems: A Study in Biologically Inspired Design
,”
Comput. Aided Des.
,
44
(
10
), pp.
879
900
.
32.
Nagel
,
J. K.
, and
Stone
,
R. B.
,
2012
, “
A Computational Approach to Biologically Inspired Design
,”
Artif. Intell. Eng. Des. Anal. Manuf
,
26
(
2
), pp.
161
176
.
33.
Deldin
,
J.-M.
, and
Schuknecht
,
M.
,
2014
, “The AskNature Database: Enabling Solutions in Biomimetic Design,”
Biologically Inspired Design
,
A.
Goel
,
D.
McAdams
, and
R.
Stone
, eds.,
Springer
,
London
, pp.
17
27
.
34.
Murphy
,
J.
,
Fu
,
K.
,
Otto
,
K.
,
Yang
,
M.
,
Jensen
,
D.
, and
Wood
,
K.
,
2014
, “
Function Based Design-by-Analogy: A Functional Vector Approach to Analogical Search
,”
ASME J. Mech. Des.
,
136
(
10
), p.
101102
.
35.
Fu
,
K.
,
Cagan
,
J.
,
Kotovsky
,
K.
, and
Wood
,
K.
,
2013
, “
Discovering Structure in Design Databases Through Functional and Surface Based Mapping
,”
ASME J. Mech. Des.
,
135
(
3
), p.
031006
.
36.
Song
,
B.
, and
Luo
,
J.
,
2017
, “
Mining Patent Precedents for Data-Driven Design: The Case of Spherical Rolling Robots
,”
ASME J. Mech. Des.
,
139
(
11
), p.
111420
.
37.
Fu
,
K.
,
Chan
,
J.
,
Cagan
,
J.
,
Kotovsky
,
K.
,
Schunn
,
C.
, and
Wood
,
K.
,
2013
, “
The Meaning of “Near” and “far”: the Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output
,”
ASME J. Mech. Des.
,
135
(
2
), p.
021007
.
38.
Luo
,
J.
,
Sarica
,
S.
, and
Wood
,
K. L.
,
2021
, “
Guiding Data-Driven Design Ideation by Knowledge Distance
,”
Knowl.-Based Syst.
,
218
, p.
106873
.
39.
Sarica
,
S.
,
Luo
,
J.
, and
Wood
,
K. L.
,
2020
, “
TechNet: Technology Semantic Network Based on Patent Data
,”
Expert Syst. Appl.
,
142
, p.
112995
.
40.
He
,
Y.
,
Camburn
,
B.
,
Liu
,
H.
,
Luo
,
J.
,
Yang
,
M.
, and
Wood
,
K.
,
2019
, “
Mining and Representing the Concept Space of Existing Ideas for Directed Ideation
,”
ASME J. Mech. Des.
,
141
(
12
), p.
121101
.
41.
He
,
Y.
,
Camburn
,
B.
,
Luo
,
J.
,
Yang
,
M. C.
, and
Wood
,
K. L.
,
2019
, “
Visual Sensemaking of Massive Crowdsourced Data for Design Ideation
,”
Proceedings of the Design Society: International Conference on Engineering Design
,
Delft, The Netherlands
,
Aug. 5–8
, Cambridge University Press, pp.
409
418
.
42.
Gonçalves
,
M.
,
Cardoso
,
C.
, and
Badke-Schaub
,
P.
,
2014
, “
What Inspires Designers? Preferences on Inspirational Approaches During Idea Generation
,”
Des. Stud.
,
35
(
1
), pp.
29
53
.
43.
Macomber
,
B.
, and
Yang
,
M.
,
2011
, “
The Role of Sketch Finish and Style in User Responses to Early Stage Design Concepts
,”
Proceedings ASME 2011 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Washington, DC
,
January
, pp.
567
576
.
44.
Linsey
,
J. S.
,
Clauss
,
E.
,
Kurtoglu
,
T.
,
Murphy
,
J.
,
Wood
,
K.
, and
Markman
,
A.
,
2011
, “
An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods
,”
ASME J. Mech. Des.
,
133
(
3
), p.
031008
.
45.
Atilola
,
O.
,
Tomko
,
M.
, and
Linsey
,
J. S.
,
2016
, “
The Effects of Representation on Idea Generation and Design Fixation: A Study Comparing Sketches and Function Trees
,”
Des. Stud.
,
42
, pp.
110
136
.
46.
McKoy
,
F. L.
,
Vargas-Hernández
,
N.
,
Summers
,
J. D.
, and
Shah
,
J. J.
,
2001
, “
Influence of Design Representation on Effectiveness of Idea Generation
,”
Proceedings of ASME DETC
,
Pittsburgh, PA
,
September
, pp.
9
12
.
47.
Casakin
,
H.
,
2004
, “
Visual Analogy as a Cognitive Strategy in the Design Process: Expert Versus Novice Performance
,”
J. Des. Res.
,
4
(
2
), p.
124
.
48.
Casakin
,
H. P.
, and
Goldschmidt
,
G.
,
2000
, “
Reasoning by Visual Analogy in Design Problem-Solving: The Role of Guidance
,”
Environ. Plann. B Plann. Des.
,
27
(
1
), pp.
105
119
.
49.
Stiny
,
G.
,
1993
, “Emergence and continuity in shape grammars,”, Proceedings CAAD Futures,
U.
Flemming
, and
S.
Van Wyk
, eds.,
North-Holland
,
Amsterdam
, pp.
37
54
.
50.
Zijian
,
Z
, and
Yan
,
J
,
2021
, “
Toward Computer Aided Visual Analogy Support (CAVAS): Augment Designers Through Deep Learning
,”
Proceedings ASME 2021 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual
,
August
.
51.
Zijian
,
Z
, and
Yan
,
J
,
2022
, “
Data-Enabled Sketch Search and Retrieval for Visual Design Stimuli Generation
,”
Artificial Intelligence for Engineering Design, Analysis and Manufacturing
,
36
, p.
e25
.
52.
Bell
,
S.
, and
Bala
,
K.
,
2015
, “
Learning Visual Similarity for Product Design With Convolutional Neural Networks
,”
ACM Trans. Graph.
,
34
(
4
), pp.
1
10
.
53.
Pu
,
Y.
,
Gan
,
Z.
,
Henao
,
R.
,
Yuan
,
X.
,
Li
,
C.
,
Stevens
,
A.
, and
Carin
,
L.
,
2016
, “
Variational Autoencoder for Deep Learning of Images, Labels and Captions
,”
Proceedings Advances in Neural Information Processing Systems
,
Barcelona, Spain
,
December
, pp.
2352
2360
.
54.
Yu
,
Q.
,
Yang
,
Y.
,
Song
,
Y.-Z.
,
Xiang
,
T.
, and
Hospedales
,
T.
,
2017
, “
Sketch-a-Net That Beats Humans
,”
Int. J. Comput. Vis.
,
122
, pp.
411
425
.
55.
Karimi
,
P.
,
Maher
,
M. L.
,
Davis
,
N.
, and
Grace
,
K.
,
2019
, “
Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System
,” arXiv preprint arXiv:1906.10188.
56.
Jiang
,
S.
,
Luo
,
J.
,
Ruiz-Pava
,
G.
,
Hu
,
J.
, and
Magee
,
C. L.
,
2021
, “
Deriving Design Feature Vectors for Patent Images Using Convolutional Neural Networks
,”
ASME J. Mech. Des.
,
143
(
6
), p.
061405
.
57.
Jiang
,
S.
,
Luo
,
J.
,
Pava
,
G. R.
,
Hu
,
J.
, and
Magee
,
C. L.
,
2020
, “
A CNN-Based Patent Image Retrieval Method for Design Ideation
,”
58.
Ha
,
D.
, and
Eck
,
D.
,
2017
, “
A Neural Representation of Sketch Drawings
,” arXiv preprint arXiv:1704.03477.
59.
Kingma
,
D. P.
, and
Welling
,
M.
,
2013
, “
Auto-Encoding Variational Bayes
,” arXiv preprint arXiv:1312.6114.
60.
Schuster
,
M.
, and
Paliwal
,
K. K.
,
1997
, “
Bidirectional Recurrent Neural Networks
,”
IEEE Trans. Signal Process.
,
45
(
11
), pp.
2673
2681
.
61.
Reed
,
S.
,
van den Oord
,
A.
,
Kalchbrenner
,
N.
,
Colmenarejo
,
S. G.
,
Wang
,
Z.
,
Chen
,
Y.
,
Belov
,
D.
, and
de Freitas
,
N.
,
2017
, “
Parallel Multiscale Autoregressive Density Estimation
,”
Proceedings of the 34th International Conference on Machine Learning
,
Sydney, Australia
,
July
, Vol. 70, JMLR, pp.
2912
2921
.
62.
Chen
,
Y.
,
Tu
,
S.
,
Yi
,
Y.
, and
Xu
,
L.
,
2017
,
Sketch-pix2seq: A Model to Generate Sketches of Multiple Categories
,” arXiv preprint arXiv:1709.04121.
63.
Goldschmidt
,
G.
,
1994
, “
On Visual Design Thinking: The vis Kids of Architecture
,”
Des. Stud.
,
15
(
2
), pp.
158
174
.
64.
Jongejan
,
J.
,
Rowley
,
H.
,
Kawashima
,
T.
,
Kim
,
J.
, and
Fox-Gieg
,
N.
,
2016
, Mount View, CA, https://quickdraw.withgoogle.com/, Accessed February 17, 2018.
65.
Xie
,
J.
,
Girshick
,
R.
, and
Farhadi
,
A.
,
2016
, “
Unsupervised Deep Embedding for Clustering Analysis
,”
Proceedings of the International Conference on Machine Learning
,
New York
,
June
, pp.
478
487
.
66.
Van der Maaten
,
L.
, and
Hinton
,
G.
,
2008
, “
Visualizing Data Using t-SNE
,”
J. Mach. Learn. Res.
,
9
(
11
), pp.
2579
2605
.
67.
Nigam
,
K.
, and
Ghani
,
R.
,
2000
, “
Analyzing the Effectiveness and Applicability of co-Training
,”
Proceedings of the Ninth International Conference on Information and Knowledge Management
,
McLean, VA
,
November
, pp.
86
93
.
68.
Kuhn
,
H. W.
,
1955
, “
The Hungarian Method for the Assignment Problem
,”
Nav. Res. Logist. Q.
,
2
(
1–2
), pp.
83
97
.
69.
Vetter
,
P.
, and
Newen
,
A.
,
2014
, “
Varieties of Cognitive Penetration in Visual Perception
,”
Conscious. Cogn.
,
27
, pp.
62
75
.
70.
Meyer
,
B. H.
,
Pozo
,
A. T. R.
, and
Zola
,
W. M. N.
,
2022
, “
Global and Local Structure Preserving GPU t-SNE Methods for Large-Scale Applications
,”
Expert Syst. Appl.
,
201
, p.
116918
.
71.
Agis
,
D.
, and
Pozo
,
F.
,
2019
, “
A Frequency-Based Approach for the Detection and Classification of Structural Changes Using t-SNE
,”
Sensors
,
19
(
23
), p.
5097
.
72.
Srinivasan
,
V.
,
Song
,
B.
,
Luo
,
J.
,
Subburaj
,
K.
,
Elara
,
M. R.
,
Blessing
,
L.
, and
Wood
,
K.
,
2018
, “
Does Analogical Distance Affect Performance of Ideation?
,”
ASME J. Mech. Des.
,
140
(
7
), p.
071101
.
You do not currently have access to this content.