Abstract

Can we provide evidence-based guidance to instructors to improve the delivery of the course based on students’ reflection on doing? Over three years at the University of Oklahoma, Norman, USA, we have collected about 18,000 Take-aways from almost 400 students who participated in an undergraduate design, build, and test course. In this paper, we illustrate the efficacy of using the Latent Dirichlet Algorithm to respond to the question posed above. We describe a method to analyze the Take-aways using a Latent Dirichlet Allocation (LDA) algorithm to extract topics from the Take-away data and then relate the extracted topics to instructors’ expectations using text similarity. The advantage of the LDA algorithm is anchored in that it provides a means for summarizing large amount of take-away data into several key topics so that instructors can eliminate the labor-intensive evaluation of it. By connecting and comparing what students learned (embodied in Take-aways) and what instructors expected the students to learn (embodied in stated Principles of Engineering Design), we provide evidence-based guidance to instructors on how to improve the delivery of AME4163: Principles of Engineering Design. Our objective in this paper is to introduce a method for quantifying text data to facilitate an instructor to modify the content and delivery of the next version of the course. The proposed method can be extended to other courses patterned after AME4163 to generate similar data sets covering student learning and instructor expectations, and the LDA algorithm can be used for dealing with the large amount of textual data embodied in students’ Take-aways.

References

1.
Mistree
,
F.
,
2013
, “
Strategic Design Engineering: A Contemporary Paradigm for Engineering Design Education for the 21st Century?
,”
ASME J. Mech. Des.
,
135
(
9
), p.
090301
.
2.
Autrey
,
J. L.
,
Sieber
,
J. M.
,
Siddique
,
Z.
, and
Mistree
,
F.
,
2018
, “
Leveraging Self-assessment to Encourage Learning Through Reflection on Doing
,”
Int. J. Eng. Educ.
,
34
(
2B
), pp.
708
722
.
3.
Turns
,
J.
,
Newstetter
,
W.
,
Allen
,
J. K.
, and
Mistree
,
F.
,
1997
, “
The Reflective Learner: Supporting the Writing of Learning Essays That Support the Learning of Design Through Projects
,”
American Society of Engineering Education
,
Milwaukee, WI
,
June 15–18
.
4.
Peng
,
S.
,
Ming
,
Z.
,
Siddique
,
Z.
,
Allen
,
J.
, and
Mistree
,
F.
,
2020
, “
Work in Progress: Quantifying Learning by Reflecting on Doing in an Engineering Design, Build and Test Course
,”
American Society for Engineering Education Annual Conference and Exposition
, Paper No. 31679.
5.
Peng
,
S.
,
Ming
,
Z.
,
Allen
,
J. K.
,
Siddique
,
Z.
, and
Mistree
,
F.
,
2020
, “
Quantification of Students’ Learning Through Reflection on Doing Based on Text Similarity
,”
ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Aug. 17–19
, Paper No. DETC2020-22267.
6.
Schon
,
D. A.
,
1984
,
The Reflective Practitioner: How Professionals Think in Action
, Vol.
5126
,
Basic Books
,
New York
.
7.
Baird
,
J. R.
,
Fensham
,
P. J.
,
Gunstone
,
R. F.
, and
White
,
R. T.
,
1991
, “
The Importance of Reflection in Improving Science Teaching and Learning
,”
J. Res. Sci. Teach.
,
28
(
2
), pp.
163
182
.
8.
Adams
,
R. S.
,
Turns
,
J.
, and
Atman
,
C. J.
,
2003
, “
Educating Effective Engineering Designers: The Role of Reflective Practice
,”
Des. Stud.
,
24
(
3
), pp.
275
294
.
9.
Van Beveren
,
L.
,
Roets
,
G.
,
Buysse
,
A.
, and
Rutten
,
K.
,
2018
, “
We All Reflect, But Why? A Systematic Review of the Purposes of Reflection in Higher Education in Social and Behavioral Sciences
,”
Educ. Res. Rev.
,
24
, pp.
1
9
.
10.
Bernard
,
A. W.
,
Gorgas
,
D.
,
Greenberger
,
S.
,
Jacques
,
A.
, and
Khandelwal
,
S.
,
2012
, “
The Use of Reflection in Emergency Medicine Education
,”
Acad. Emerg. Med.
,
19
(
8
), pp.
978
982
.
11.
Dutson
,
A. J.
,
Todd
,
R. H.
,
Magleby
,
S. P.
, and
Sorensen
,
C. D.
,
1997
, “
A Review of Literature on Teaching Engineering Design Through Project-Oriented Capstone Courses
,”
J. Eng. Educ.
,
86
(
1
), pp.
17
28
.
12.
Shergadwala
,
M. N.
,
Panchal
,
J. H.
, and
Ramani
,
K.
,
2019
, “
Students’ Decision-Making in a Product Design Process: An Observational Study
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
, Paper No. DETC2019-98216.
13.
Shah
,
D.
,
Kames
,
E.
,
Clark
,
M.
, and
Morkos
,
B.
,
2019
, “
Development of a Coding Scheme for Qualitative Analysis of Student Motivation in Senior Capstone Design
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
, Paper No. DETC2019-98423.
14.
Joshi
,
S.
, and
Summers
,
J. D.
,
2011
, “
A Coding Scheme for Analyzing Capstone Design Reports: Problem and Solution Descriptions
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Washington, DC
, Paper No. DETC2011-47154.
15.
Riegel
,
C.
,
Starkey
,
E. M.
,
Hunter
,
S. T.
, and
Miller
,
S. R.
,
2019
, “
Do Students Want to Dissect?: A Survey of Student Opinions on the Use of Product Dissection in the Section Room
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
, Paper No. DETC2019-97569.
16.
Alsager Alzayed
,
M.
,
McComb
,
C.
,
Hunter
,
S. T.
, and
Miller
,
S. R.
,
2019
, “
Expanding the Solution Space in Engineering Design Education: A Simulation-Based Investigation of Product Dissection
,”
ASME J. Mech. Des.
,
141
(
3
), p.
032001
.
17.
Bal l
,
Z.
,
Bessette
,
J.
, and
Lewis
,
K.
,
2020
, “
Who, What, and When? Exploring Student Focus in the Capstone Design Experience
,”
ASME International Design Technical Conferences, Design Education Conference
,
St. Louis, MO
, Paper No. DETC2020-22027.
18.
Born
,
W.
, and
Schmidt
,
L.
,
2018
, “
Evaluating the Project Activity Differences in Capstone Design Students via Journals
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Quebec, Canada
, Paper No. DETC2018-85876.
19.
Maier
,
J. R.
,
Troy
,
T.
,
Johnston
,
P. J.
,
Bobba
,
V.
, and
Summers
,
J. D.
,
2010
, “
Case Study Research Using Senior Design Projects: An Example Application
,”
ASME J. Mech. Des.
,
132
(
11
), p.
111011
.
20.
Ruder
,
J.
, and
Sobek
,
D. K.
,
2005
, “
Student System Level Design Activities: An Empirical Pilot Study on Improving Design Outcomes
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Long Beach, CA
, Paper No. DETC2005-85281.
21.
Suwa
,
M.
,
Purcell
,
T.
, and
Gero
,
J.
,
1998
, “
Macroscopic Analysis of Design Processes Based on a Scheme for Coding Designers’ Cognitive Actions
,”
Des. Stud.
,
19
(
4
), pp.
455
483
.
22.
Sobek
,
D.
,
2002
, “
Preliminary Findings From Coding Student Design Journals
,”
Proceedings of the 2002 American Society for Engineering Education Annual Conference and Exposition
.
23.
Atman
,
C. J.
,
Adams
,
R. S.
,
Cardella
,
M. E.
,
Turns
,
J.
,
Mosborg
,
S.
, and
Saleem
,
J.
,
2007
, “
Engineering Design Processes: A Comparison of Students and Expert Practitioners
,”
J. Eng. Educ.
,
96
(
4
), pp.
359
379
.
24.
Salton
,
G.
, and
McGill
,
M. J.
,
1986
,
Introduction to Modern Information Retrieval
,
McGraw-Hill
,
New York, NY
.
25.
Beel
,
J.
,
Gipp
,
B.
,
Langer
,
S.
, and
Breitinger
,
C.
,
2016
, “
Research-Paper Recommender Systems: A Literature Survey
,”
Int. J. Digit. Libr.
,
17
(
4
), pp.
305
338
.
26.
Blei
,
D. M.
,
Ng
,
A. Y.
, and
Jordan
,
M. I.
,
2003
, “
Latent Dirichlet Allocation
,”
J. Mach. Learn. Res.
,
3
(
Jan.
), pp.
993
1022
.
27.
Blei
,
D. M.
,
2011
, “
Introduction to Probabilistic Topic Models
,”
Commun. ACM
,
55
(
4
), pp.
77
84
.
28.
Ahmed
,
F.
,
Fuge
,
M.
, and
Gorbunov
,
L. D.
,
2016
, “
Discovering Diverse, High Quality Design Ideas From a Large Corpus
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
, Paper No. DETC2016-59926.
29.
Blei
,
D. M.
, and
Lafferty
,
J. D.
,
2007
, “
A Correlated Topic Model of Science
,”
Ann. Appl. Stat.
,
1
(
1
), pp.
17
35
.
30.
Rosen-Zvi
,
M.
,
Griffiths
,
T.
,
Steyvers
,
M.
, and
Smyth
,
P.
,
2004
, “
The Author-Topic Model for Authors and Documents
,”
Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence
,
Banff, Canada
,
July 7–11
, AUAI Press, pp.
487
494
.
31.
Yau
,
C.-K.
,
Porter
,
A.
,
Newman
,
N.
, and
Suominen
,
A.
,
2014
, “
Clustering Scientific Documents With Topic Modeling
,”
Scientometrics
,
100
(
3
), pp.
767
786
.
32.
Ayoub
,
J.
,
Zhou
,
F.
,
Xu
,
Q.
, and
Yang
,
J.
,
2019
, “
Analyzing Customer Needs of Product Ecosystems Using Online Product Reviews
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
, Paper No. DETC2019-97642.
33.
Ball
,
Z.
, and
Lewis
,
K.
,
2019
, “
Predicting Multi-Disciplinary Design Performance Utilizing Automated Topic Discovery
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
, Paper No. DETC2019-97189.
34.
Ball
,
Z.
, and
Lewis
,
K.
,
2020
, “
Predicting Design Performance Utilizing Automated Topic Discovery
,”
ASME J. Mech. Des.
,
142
(
12
), p.
121703
.
35.
Johri
,
A.
,
Wang
,
G. A.
,
Liu
,
X.
, and
Madhavan
,
K.
,
2011
, “
Utilizing Topic Modeling Techniques to Identify the Emergence and Growth of Research Topics in Engineering Education
,”
2011 Frontiers in Education Conference (FIE)
,
Rapid City, SD
,
Oct. 12–15
, IEEE, pp. T2F-1–T2F-6.
36.
Dong
,
A.
,
Hill
,
A. W.
, and
Agogino
,
A. M.
,
2004
, “
A Document Analysis Method for Characterizing Design Team Performance
,”
ASME J. Mech. Des.
,
126
(
3
), pp.
378
385
.
37.
Dong
,
A.
,
2005
, “
The Latent Semantic Approach to Studying Design Team Communication
,”
Des. Stud.
,
26
(
5
), pp.
445
461
.
38.
Kleinsmann
,
M.
,
Buijs
,
J.
, and
Valkenburg
,
R.
,
2010
, “
Understanding the Complexity of Knowledge Integration in Collaborative New Product Development Teams: A Case Study
,”
J. Eng. Technol. Manage.
,
27
(
1–2
), pp.
20
32
.
39.
Kakkonen
,
T.
,
Myller
,
N.
, and
Sutinen
,
E.
,
2006
, “Applying Latent Dirichlet Allocation to Automatic Essay Grading,”
Advances in Natural Language Processing. FinTAL 2006. Lecture Notes in Computer Science
, Vol.
4139
,
T.
Salakoski
,
F.
Ginter
,
S.
Pyysalo
, and
T.
Pahikkala
, eds.,
Springer
,
Berlin/Heidelberg
.
40.
Perez-Marin
,
D.
,
2009
,
Adaptive Computer Assisted Assessment of Free-Text Students’ Answers: An Approach to Automatically Generate Students’ Conceptual Models
,
VDM Verlag
,
Saarbrücken, Germany
.
41.
Xing
,
W.
,
Lee
,
H.
, and
Shibani
,
A.
,
2020
, “
Identifying Patterns in Students’ Scientific Argumentation: Content Analysis Through Text Mining Using Latent Dirichlet Allocation
,”
Educ. Technol. Res. Dev.
,
68
(
5
), pp.
2185
2214
.
42.
Ming
,
N. C.
, and
Ming
,
V. L.
,
2012
, “
Predicting Student Outcomes From Unstructured Data
,”
CEUR Workshop Proceedings
, Vol. 872.
43.
Ganapathy
,
C.
,
Kang
,
J. H.
,
Shaw
,
E.
, and
Kim
,
J.
,
2011
, “Sectionification Techniques for Assessing Student Collaboration in Shared Wiki Spaces,”
Artificial Intelligence in Education. AIED 2011. Lecture Notes in Computer Science
, Vol.
6738
,
G.
Biswas
,
S.
Bull
,
J.
Kay
, and
A.
Mitrovic
, eds.,
Springer
,
Berlin/Heidelberg
.
44.
Jordan
,
M. I.
,
1998
,
Learning in Graphical Models
,
Springer Science & Business Media
,
Dordrecht
.
45.
Hoffman
,
M.
,
Bach
,
F. R.
, and
Blei
,
D. M.
,
2010
, “
Online Learning for Latent Dirichlet Allocation
,”
Advances in Neural Information Processing Systems
,
Vancouver, British Columbia, Canada
,
Dec. 6–9
, pp.
856
864
.
46.
Wu
,
Y.
,
Ming
,
Z.
,
Allen
,
J. K.
, and
Mistree
,
F.
,
2022
, “
Evaluation of Students’ Learning Through Reflection on Doing Based on Sentiment Analysis
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
St. Louis, MO
, Paper No. DETC2022-88161.
47.
Cer
,
D.
,
Yang
,
Y.
,
Kong
,
S.-Y.
,
Hua
,
N.
,
Limtiaco
,
N.
,
John
,
R.
,
Constant
,
N.
, et al
,
2018
, “
Universal Sentence Encoder for English
,”
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
,
Brussels, Belgium
,
Oct. 31–Nov. 4
, pp.
169
174
.
You do not currently have access to this content.