Framework for the Evolution of Heuristics in Advanced Manufacturing

Heuristics are context-dependent actions taken by designers in hopes of reaching “satisfactory but not necessarily optimal” solutions [1]. These actions range from general strategies that guide the design process, down to detailed decisions about the artifact. Koen emphasizes heuristics as decisions that are prone to error and do not always produce one’s desired outcomes, emphasizing their dependence on context for success [2]. Following the heuristic representation format provided by Binder, the authors use a context–action pairing to present heuristics in this study, as well as in prior studies [3–5]. For example, “if the material is over/under building (context), slow down/speed up the feed (action).”

Research toward understanding heuristics largely began with Kahneman and Tversky, when they presented contexts where humans do not approach decision-making in the same manner as popular decision-making models, namely, utility theory [6–9]. In an attempt to justify the use of heuristics, Gigerenzer later pushed for heuristic research that described not only the heuristic being used but the contexts in which they were successful or unsuccessful [10]. From the perspective of human evolution, researchers have also attempted to justify the use of heuristics through “adaptive rationality” [11,12]. Adaptive rationality is based on the theory that cognitive biases are derived from humans’ evolutionary will to survive and are not weaknesses or errors, but rather efficient adaptations of the mind to enable survival. This body of work divides cognitive biases into three types: heuristics, error management effects, and experimental artifacts. Error management refers to making decisions toward less costly errors (preferring the cost of a false-positive over the cost of a false-negative). Experimental artifacts are the result of research strategies or designs that place humans in unnatural settings or apply inappropriate norms. It is noted that these types are not mutually exclusive, as biases may fall into more than one category.

This study focuses on the knowledge gap in understanding how heuristics are perceived by designers, as well as how they are developed through designers’ experiences. There is also an emphasis on improving heuristic extraction and characterization methodology. Most of the previous literature studied in design heuristics has used artifact analysis as the primary method for heuristic extraction, where an artifact is defined as any tangible object that can be presented physically or through images/sketching [13–23]. This form of study typically begins with identifying features within a set of artifacts/products and then hypothesizing the respective actions resulting in those features. As a case study, data were collected using in this research using multiple methods: document analysis, interviews, and surveys. While some prior work has used these methods before, this is the first known work that implements all three methods in series [24–32]. Lastly, this study aims to expand understanding of how these heuristics are implemented from an error management perspective.

Accuracy in the documentation of heuristics is beneficial for translating them to new team members and other training processes. This accuracy is especially critical as a foundation for new manufacturing technology in current development. In regard to manufacturing, prior work has been produced in testing the implementation of heuristics that guide a Design for Additive Manufacturing (DfAM) context [33–35]. The methods in this study extract heuristics from participants in positions requiring specific advanced manufacturing technology still in development. The heuristics in this study were labeled as applicable to many stages, ranging from part design and build planning to machine setup and quality assessment. As such, the heuristics may apply to both designers and machine operators. Five participants were familiar with the Mazak VC500, a hybrid machine combining additive manufacturing and computerized numerical control (CNC) technology [36]. The additive process is a directed energy deposition (DED) method, in which an energy source welds the deposited material to a substrate [37]. Participants engaged in this study with respect to two versions of this technology: one that uses powder-fed deposition, and one that uses hot-wire deposition of metal. Three additional participants were familiar with the EOS M280 laser powder bed fusion (LPBF) additive manufacturing machine [38,39].

While many of the heuristics presented are process focused, there are examples of notes for understanding how hybrid manufacturing can modify one's approach to design for manufacturing. For example, consider the DfAM heuristic, “if there is an overhang on the part, ensure that the angle is smaller than 40 deg” [40]. With hybrid manufacturing, the ability to use additive and subtractive processes may eliminate this design constraint. Based on the results of this study, a new heuristic may be, “to mitigate large overhangs, consider printing a vertical wall, then machining away the extra material.” Another option may be, “to mitigate large overhangs, consider multi-axis rotation to produce the part.” Therefore, it is beneficial to begin understanding how hybrid manufacturing is currently operated on the road toward new design heuristics.

This study provides significance and originality in both the extraction process and the characterization of heuristics. One goal of this research is to move toward a more triangulated approach to documenting heuristics. For this reason, the authors use three methods in combination which have not been previously shown in known prior literature and assess the benefits and limitations of this combination. Second, it uses statistical significance to show how certain characteristics of heuristics may correlate to their perceived value, such as their perceived reliability. Lastly, it presents the first known discussion on viewing heuristics in design from a view of error management. This opens new avenues for studying heuristic decision-making justification and rationale.

This work is impactful toward a successful prescriptive research phase, as reliable extraction methodologies will lead to reliable descriptions of how designers use heuristics. Moving forward, the described heuristics and their attributes may be used to move toward more normative decision-making, ensuring that the designer chooses the heuristic that maximizes the value of their process. Heuristic competence is necessary because it gives designers the confidence needed to explore new problem spaces [41]. By understanding various aspects of how heuristics are developed and implemented, this study ultimately contributes to advancing heuristic competence within designers. To that end, with this study, the following research questions are addressed: How do designers perceive their heuristics as they develop in advanced manufacturing? What aspects of heuristics and design environments should be considered during documentation of heuristics for a repository? How might the methodology improve for heuristic extraction and characterization?

The framework presented in this study is a methodology for extracting and characterizing heuristics as they evolve in advanced manufacturing. This framework is shown in Fig. 1 and discussed in more detail throughout this section. There are several reasons for the order of the methods followed in this framework. First, the journal process allows participants to better reflect on their strategies before entering an interview session. It allows them however long they need to best put their process into words. This may make interview sessions more effective compared to participants entering a timed interview session with no preparation. The journal sessions also allow for participants to reflect using familiar terminology. It is likely that participants do not naturally use the term heuristic when thinking about their process. With the first pass at heuristic extraction, the interviewer may directly ask about specific heuristics rather than having to develop them from scratch in the interview setting. This may allow for better use of the semi-structured interview format.

This study includes eight participants, including seven graduate students in a manufacturing research lab at a major university, and one recent graduate from this university now employed at a national lab. There were seven men and one woman who participated in this study. Six participants classified themselves as white, with the other two classifying themselves as Asian. Seven participants were aged 21–30 years old, with the remaining participant aged 31–35 years old. Participants averaged 4.4 ± 1.6 years of design experience and 3.8 ± 1.85 years of manufacturing experience. For both categories, the highest amount of experience was 6 years and the lowest was 2 years of experience. At the time of the study, participants had been in their current positions an average of 2.4 ± 1.03 years, with a max of 4.5 years and a minimum of 1.5 years. All participants had some form of graduate-level education in mechanical engineering: two participants had obtained their doctorate, and five had obtained at least a master’s degree, with one participant still pursuing their master’s.

This was a valuable subject pool with which to study heuristic development due to their need to create new heuristics and refine their current heuristics to ensure successful builds and satisfactory part quality. This includes everything from designing the part for the machine, to planning the build, troubleshooting the build and assessing the part quality post-build. This knowledge is crucial to passing on to team members, novices, and for their own continuous improvement in maximizing the value of their design process. These heuristics serve as a foundation for translating design processes to newer technologies as well, as the advanced manufacturing technology evolves over time.

The methods for this study consisted of a series of two journal entries, two interviews, and one survey for each participant. The journals and interviews occurred over the course of several months, as time was needed for both participants and researchers to produce a successful study. This provided participants time to think through each journal question without significant restrictions on the time allotted for completion. After design journals were completed, time was needed for researchers to produce a preliminary heuristic extraction and scheduling of the interviews based on the availability of each participant. Additional time was needed after interview completion for heuristic refinement and for surveys to be customized for the heuristics of each participant.

Journal responses were requested and delivered securely online. Participants were asked to document aspects of their process through a set of prompted questions. Journal 1 asked for responses to a series of ten questions. The purpose of this journal was to allow participants to provide how they perceive their general process from beginning to end, regarding their interaction with the manufacturing machine to produce their desired parts.

Questions 1–2 were meant as easy questions to get the participants thinking about their process [42]. Question 1 asked which machine they would be doing the journal for, and Question 2 asked participants to list what they believe are the most important parameters and settings for their machine.

As it is possible that the participants did not normally describe their actions as “heuristics,” the journal focused on using simple, familiar language and avoiding more formal definitions of heuristics [42]. For example, Questions 3–5 asked participants for the “processes/strategies/actions” taken before using the machine, while using the machine, and any exceptions where their listed steps would not be followed. Question 7 asked for “lessons learned” while using their respective machine and for the participant to imagine if they were teaching someone else how to have success with their machine. Lastly, Question 8 asked for “rules of thumb” applied to their designs when considering their specific machine.

The journal was also set up to prompt the participants to view their actions from multiple perspectives. For example, many of the design “rules of thumb” (Question 8) may likely be applied “before using the machine” (Question 3). Question 6 asked for examples of troubleshooting, which may include actions that the participants did not initially consider in the process or process exceptions. Question 9 asked how they determine the part quality and build success, and Question 10 asked participants to list what rules of thumb they use that they have taken from experience with previous machines.

After the first journal was complete, participants moved to Journal 2. Journal 2 asked about the participants’ most recent build in an additional set of ten questions. The importance of this journal is to once again provide a different perspective for participants to reflect on their process. Considering a very specific machine use, as opposed to generalizing over many instances, may provide new contexts or actions that were not previously considered in Journal 1. This follows additional guidance from Krosnick and Presser to begin asking general questions about a topic before asking specific and targeted questions on the topic at hand [42].

Journal 2 began with five questions that assisted participants in recalling information about their most recent build. These five questions asked for the overall objective, the material used, geometry and rough dimensions, and the values for the key parameters and settings for this build. Questions 6–7 asked participants if any strategies used to plan or perform this build differed from what was documented in Journal 1. Question 8 asked participants to describe any troubleshooting that took place. This was asked to understand if potential troubleshooting issues were not documented fully in Journal 1. Questions 9–10 asked for the results of the build and any insights derived that may impact future builds. This question hoped to find the knowledge that participants may be using to develop heuristics for future builds.

After both journals were submitted, a first pass was taken at extracting the heuristics in context–action form. These heuristics were taken into the interviews for additional refinement in collaboration with the participants. Two interviews, one-hour each, were performed virtually through Microsoft Teams format and audio/video recorded, then transcribed. One researcher conducted all interviews, using a script to prompt participants to talk about their set of heuristics. Through the semi-structured interview format, the interviewer was given the freedom to focus the questioning on aspects considered important to add more clarity to the heuristic, how it was formed, or the justification for its use [43]. However, the format for each interview can be broken down into the following three different sections.

The first section of the interview was dedicated to explaining the purpose of the study and helping the participant understand why they were completing these design journals. This was followed by a more formal definition of a heuristic, as previously defined by Fu et al. [1]. The purpose was to relay to the participant why the information asked for in the design journals could be considered heuristic information, which the researchers then rephrased into context–action form.

The second section was the largest portion of the interview sessions. At this point, the interviewer presented the participants with their first pass at developing context–action heuristics from the design journals. A series of questions were then asked to the participant in an effort to get more insight into the heuristic and to improve its presentation. The questions were asked the participant to:

• Decide whether the heuristic is an accurate depiction of how they perceive their process;

• Provide an explanation for choosing this heuristic in their process;

• Provide how this heuristic came to be in their process;

• Provide any alternative actions that could have been taken; and

• Provide any key criteria that may be considered when choosing this heuristic.

Several follow-up questions were asked as needed by the researcher, using the semi-structured format. This was performed for heuristics found across all design journal data, starting with the processes before and during the use of the machine, and moving on toward heuristics in regard to troubleshooting, lessons learned, design rules, and part quality.

A final portion of the interview lasted about 15 min per participant at the end of the second interview. These questions asked participants to speak more generally about other aspects that may have an impact on how heuristics are formed or documented. Participants were asked to:

• Describe how experience with other machines impacted their process;

• Describe how they currently document their heuristic knowledge;

• Describe the areas where they wished they had more strategies or intuition; and

• Describe how their process has been impacted by: advisors/supervisors, team members, formal education, and industry standards.

After interviews, heuristics were modified as necessary for survey creation and distribution. Surveys were then distributed online through Qualtrics and gathered through web-based submission. The survey was broken into two sections. The first section asked for demographic information, such as age, gender, degrees earned, manufacturing/design experience, and how long they have been in their current position. The second section began with a Likert scale confirmation of heuristics taken from each individual’s journal and interview data. Then, additional characterizations of the heuristics were requested, such as:

• Origin of heuristic and the process stages in which they are applicable;

• How often and how long each heuristic has been used;

• Reliability and evolution of the heuristic in their process;

• Additional factors contributing to whether they choose to implement the heuristic;

• Reasons why the heuristic helps maximize the value of their process;

• Additional descriptions of how they view their heuristics.

Several guidelines from Krosnick and Presser were followed in the development of this survey [42]. For example, the survey begins with questions of low difficulty. Participants were also asked one question at a time, with similar aspects, such as origin of the heuristic, applicable stages of their design process, etc. grouped. Surveys used wording familiar to participants, and heuristics were specific to each participant. Having processes tailored to each individual should increase motivation to fill out the surveys as accurately as possible. Lastly, it is possible that this method has reduced some recall errors, as participants have already begun reflecting on their process in the journal and interview process.

With the qualitative data collected, a process called coding is performed, where patterns are found in the data for analysis [44]. Similar to previous studies, the coding process began with matching context and actions together within the design journals [4]. The interview process clarified or refined those context-action pairings, or added new contexts and actions to the set. For many aspects of the design journals, there was a clear enough context–action pairing to set up a preliminary heuristic for the interview sessions. An example of this extraction process is shown as follows.

Journal Question #3: Document any planning processes/strategies/actions that you typically go through before using this machine. Please be as thorough as possible.

Participant Response: “Most of the hybrid components we manufacture at **** are done using a CAM software called HyperMILL. The first thing to do is to import your CAD model into the software (Or design it using HyperMILL’s native CAD).”

Extracted Heuristic: Before using the machine, first import your CAD model into HyperMILL (or design it using HyperMILL's native CAD).

These heuristics formed through the journals were then discussed in more detail in the interview session. The interviewer asked questions as they considered necessary to uncover more relevant information about each action. In this example exchange shown below, enough new insight was found to refine the preliminary heuristic, while adding two new heuristics to the set.

Interviewer: “Is there any benefit to you for choosing to design it in the native CAD versus doing it in another program yourself and then importing it?”

Participant: “There is benefit to doing it in the native CAD/CAM software. So, it kind of depends on how competent each user is. You know, a lot of people know SolidWorks/Fusion, so they feel a lot more comfortable drawing parts up there. But you do it in the native CAD/CAM space when you're using the same program to design the part and do the toolpath planning. If you don't need to move the part, so you're not trying to transcribe data, you don't have to import/export models. You keep track of features and surfaces and faces, so you know like when you take a stereolithography (STL) model, if you if you start in SolidWorks, you may export something as an STL. And STL is just a mesh file, so you lose some of the native features that you designed in SolidWorks. You can also just make changes on the fly. So, let's say you're designing apart with a certain cylindrical feature. If you need to change the diameter of that feature, you can just do it in the same program. You don't have to start over from scratch, change it, export it, import it, do the toolpaths. That would be the main reason.”

Interviewer: “So ultimately, if you do it in a different place such as SolidWorks or Fusion, you're likely going to make some additional changes because things have gotten lost in translation.”

Participant: “Exactly. You have data loss.”

Interviewer: “…And do those other CAM packages typically work well with the Mazak?”

Participant: “It depends on what you're doing…In the world that I deal with in hybrid manufacturing, not every CAM package offers additive manufacturing. And each package kind of has a different level of expertise. So, for example, Fusion: if I was doing a very simple geometry that only requires three axes, you know very something very simple, I would use Fusion. I can do it quicker and easier. But if I'm doing something very complex, I would want to go into Hyper Mill. It's a little more robust, but it's not as user friendly.”

Refined Heuristics:

1. When developing your CAD model, use the native CAD package for ease of editing and to avoid data loss through importing the model.

2. When working with simple geometries, use Fusion to develop the CAD model quicker/easier.

3. When working with complex geometries, use Hyper Mill for a more robust CAD model development.

When the interviewer presented a heuristic to the participant, the participant most often verbally agreed that the heuristics were a part of their process. Therefore, most heuristics were only modified through the additional lines of questioning. However, in some instances, the participant realized that some information was missing. In this example, the participant realized that an “essential” step was not included in their design journal.

Journal Question #3: Document any planning processes/strategies/actions that you typically go through before using this machine. Please be as thorough as possible.

Participant Response: Mazak Hybrid System—The first step is to set up a work coordinate system for the substrate, I would be printing on.

Preliminary Heuristic: Before using the Mazak machine, the first step is to set up a work coordinate system of the substrate used for printing.

Interviewer: “So, the first thing you said was before using the Mazak machine, the first step is to set up a work coordinate system of the substrate used for printing. Does that sound like an accurate depiction of how you perceive that part of your process?”

Participant: “Yeah, I did leave out one thing—I guess picking out the workpiece as well would be an essential step as well. Like cutting it to the proper size and selecting what material that you’re wanting to use, but that’s going to depend on what you want to print. So, like if you have this material, print on this workpiece, for example.”

Interviewer: “Ok and that depends on the material that you are going to use.”

Participant: “Correct.”

Interviewer: “Could you explain why it’s necessary from the machine standpoint to set up with the work coordinate system with your workpiece.”

Participant: “So the main importance is to make sure that you align, and especially with hybrid, that you align your part with your additive toolhead and your subtractive toolhead. The thing is that the additive toolhead is actually offset with the machining toolhead…you can potentially print in an entirely different area than you wanted to if you just used someone else’s work coordinate system. Worst case is you actually crash into the part. So, it’s very important to do the work coordinate system first and set up your workpiece into the system.”

Extracted Heuristics:

1. Before using the Mazak machine, the first step is to set up a work coordinate system of the substrate to align your tool heads.

2. When selecting the workpiece, choose the substrate material based on the material being used for printing.

The connection between contexts and actions was not always clear in the design journals. In some cases, the participant would write a phrase with no clear direction. Consider the rule of thumb from one participant below, followed by the interview exchange. The interviewer had to probe the participant as necessary without an initial heuristic present.

Journal Question #8: List your most common rules of thumb that you apply to your designs when designing parts for fabrication on this machine.

Participant Response: Layer height of the beads.

Interviewer: “You wrote a line that I wanted a little more clarity on, you just said ‘layer height of the beads.’ I wasn't sure exactly what you were referring to?”

Participant: “OK, so what layer height is, say this is one bead…You'll notice that usually the first layer actually has a different layer height then the other heights as you build more…So what we do is we just take an average and get the average layer height per bead. So, this would be one layer, two layers and three layers.”

Interviewer: “Do you need that value as something critical to produce the build?”

Participant: “It's definitely necessary because as you build taller you do not have the proper layer height. Say for example you have too low of a layer height, so it's actually smaller than it should be…What happens is as you build taller, this distance gets further and further, so your parameters will actually change and vice versa. So, you have too much of a layer height, so you predicted that if it's too tall, like say you have a very large layer height then actually what will happen is this will grow taller. But if it's like too small of a layer height this will actually get closer and closer until you might hit collision into your part. So ideally, if you can get a proper layer height as you build it, you should have a consistent distance between your laser and your printed bead.”

Interviewer: “Is that something that you can calculate beforehand, or something that you need to run a few lines before you know what that's going to be?”

Participant: “Yeah, it's not something you could calculate.”

Extracted Heuristic: When setting the layer height of the beads, use an average of the first few layers to account for height differences.

The goal of the journal and interview process was to produce a quality set of extracted heuristics which can then be characterized through the survey method. The results section will focus on the results and analysis of survey data. The discussion section will overview any other insight towards heuristics found during the interviews, as well as an assessment of the journal and interview processes. The full set of heuristics can be found in the  Appendix in Tables 1 and 2.

After the journal and interview extraction process, participants confirmed in the survey whether the resulting heuristics were used in their own process. This was a Likert scale response shown in Fig. 2. Of 126 heuristics, only four were listed as “somewhat disagree,” and no heuristics were listed as “strongly disagree.” These four heuristics have since been taken out of the additional study survey analysis. It is unclear how these were invalid heuristics, as there was no follow-up discussion as part of the research study. It can be noted that three heuristics were taken from journal responses that required additional information to develop the heuristic, similar to “Example Three” in the previously described heuristic extraction process.

Figure 3 shows participants overwhelmingly listed experience as the main form of heuristic development, which strengthens our definition of a heuristic. It is also not surprising that colleagues provided almost 40% of heuristics given, as these heuristics were developed from the experiences of others in similar contexts. It is also sensible that academic literature appeared in higher numbers than a textbook or educational origins, as the participants use these machines for research purposes, and new research is being published consistently on the machine technology.

Because 75% of responses included originating from experience, experience is broken down in relation to the other sources as well, as shown in Fig. 4. It is shown that most heuristics were not listed as just experience only, but from other sources as well. The largest of these combinations was experience plus knowledge from a colleague, followed by experience plus machine-specific training. It is possible that while participants were initially given heuristics from other people, they did not consider them part of their own process until they saw the success of those heuristics in their own experiences. For 91 heuristics where the experience was listed as an origin, 60 (66%) were connected to “past failures” contributing to their decision to implement that heuristic. Similarly, of 32 heuristics where the experience was the only listed origin, 19 (59%) were connected to “past failures” contributing to their decision to implement that heuristic.

Figure 5 shows the stages in which the participants labeled their heuristics as applicable. These stages were given in the survey based on the processes discussed in the journals and interviews. It is reasonable that “during build” is the lowest category, as there is little to do for most participants outside of listening and watching for things out of the ordinary. Similarly, the action items after the part have been removed from the machine (post-processing, quality assessment) should be smaller compared to the amount of planning, design, and setup required before machine use, which sets the designer up for the best results on the other end of machine use. Troubleshooting heuristics accounting for more than 20% of heuristics may show how much participants rely on experience to develop heuristics. It may also have implications for the types of information that participants were able to easily recall; it is possible that failures are easily retained or that participants retain troubleshooting heuristics well due to the importance of proper maintenance and function of the machines to avoid repair costs. As shown in Fig. 6, most troubleshooting heuristics are based on past failures.

Figure 7 shows how long heuristics have been used by their respective users. No one put “unsure” for this question, although one participant did skip this question for one heuristic. As stated in the demographics section, participants averaged 4.4 ± 1.6 years of design experience and 3.8 ± 1.85 years of manufacturing experience. Most of the heuristics presented here have been used for 1–2 years. This falls closer to the duration that participants reported having been in their current positions, an average of 2.4 ± 1.03 years, with a max of 4.5 years and a minimum of 1.5 years. It is possible that participants started these positions with only a small set of heuristics that translated and built their toolbox over time.

In Fig. 8, responses are shown for which participants checked the ways in which the heuristic maximized their process. They were able to choose from eight factors presented in the survey. The results indicate that participants are more concerned about meeting the requirements of the part than saving time and resources. Another way of seeing this could be that participants see their efficiency in terms of preventing machine failures, rather than successfully saving time, material, or other resources.

The contrast between process efficiency and part quality could also potentially be explained across machine users. When comparing EOS to Mazak, 56% of EOS heuristics (27 of 48 heuristics) were characterized as “prevents machine failure,” compared to 43% of Mazak heuristics (32 of 74 heuristics). Only ten heuristics were perceived as meeting both perspectives of efficiency: saving the participant’s both time and material/resources. Nine of these ten heuristics were delivered by the three EOS participants. This may be interesting in terms of placing value on a “good” heuristic. Being efficient on multiple levels may provide a safety net for using the heuristic and still having success. For example, if the heuristic does not save time during one build, it may still provide efficiency in terms of the material used.

Figure 9 shows a set of seven factors participants could choose from, which describe aspects contributing to the participant’s decision to implement the heuristic. Half of the heuristics are associated with input from team members before implementing those actions. Based on interview responses in section three of the interview, all participants explain that they learned the most through guessing and checking and would talk to team members, advisors, or supervisors to get advice and a general understanding of the machine. Participants trust these heuristics because of their background experience. As participant P8 explains, there is no reason to not trust strategies from other team members because those members had more experience in that area: “In the absence of any knowledge of it, I guess I have no reason to suspect any of it…especially coming in here to (redacted), I would definitely, you know, trust whatever anybody said because I had no experience with it at all…it depends on whether I know anything about the subject or not.”

Figure 10 shows more characteristics that participants were asked to assign to their heuristics if applicable. Understandably, the proactive description was associated with more heuristics than the reactive description, as participants consistently implied wanting to avoid crashes or failures that would restart the build or machining process. Only six heuristics in total were characterized as risky, and they were all from Mazak VC 500 users. Several of these “risky” heuristics were related to participants making intuitive judgments in the middle of a build:

• For better part quality, run the nozzle closer to the part.

• If the material is over/under the building, slow down/speed up the feed.

• If the build makes noise due to significant overbuilding, manually slow down the feed rate and deposit more material in lower areas to even out the part.

These may have been considered risky because they are relying more on their own intuition on a case-by-case basis. They must trust themselves to hear the right noise, manipulate the machine to the right speed, sense where the nozzle works best, etc. This may come with high rewards, but with the risk of failure that requires a restart. This is interesting because one way of looking at heuristics is that a “good” heuristic is also “safe” because it is used to produce a satisfactory outcome. It is possible that the participants using “risky” heuristics may not be aware of additional “safe” heuristics to use at this point in their experience level.

Only around 40% of heuristics were noted as easy to recognize the context to apply the heuristic. This visualizes the idea of designers having a heuristic versus knowing when to use it. The participants of this study understand which heuristics they apply, although they still find difficulty in understanding when to implement them. Lastly, as over half of the heuristics were described as performed implicitly, it is possible that participants were able to consider more implicit heuristics during the journal and interview process. The journal method gave participants sufficient time to consider their whole process and from multiple perspectives, and the interview asked them to assess why they made those decisions.

Figures 11–13 show how participants described the heuristics in terms of reliability, frequency of use, and evolution. There were no participants who put “unsure” to “How often does this heuristic evolve?” However, one heuristic failed to receive a completed survey response for each of the three attributes. This certainty in responses may be due to the interviewer constantly asking participants to reflect on how their actions have evolved while using their respective machines. These three survey questions were correlated using Spearman’s correlation, with a discussion of them following the figures.

Figure 14 shows the combination of responses for heuristic reliability and its frequency of use. There was a significant positive correlation between the reliability of a heuristic and its frequency of use (Spearman’s ρ = 0.538, p < 0.001, N = 121). This means that as the perceived reliability of the heuristics increases, it tends to be used more in the participant’s process. Consider the examples below. One reason that increasing the layer size is considered low reliability is because there may be better actions available for the specific case, such as changing the recoater blade type. For a highly reliable action, such as setting the build order from bottom left to top right, this is driven by a machine phenomenon that will be consistent from build to build.

Low Frequency, Low Reliability: If the recoater blade crashes, consider increasing your layer size to decrease the chances of another crash.

High Frequency, High Reliability: When setting the build order, build from bottom left to top right to minimize the impact of metal condensate.

Figure 15 shows the combination of responses for heuristic reliability and its evolution. There was a significant negative correlation between the reliability of a heuristic and its evolution (Spearman’s ρ = −0.437, p < 0.001, and N = 121). This means that heuristics considered to be more reliable are also perceived as changing less often. Consider the examples below. It’s possible that several factors contribute to the heuristic being less reliable and changing often. These could include the participant modifying which colors they believe have resulted in better parts, inconsistency in color being a true correlation to material properties or the criteria for the quality of specific parts fluctuating. For the more reliable heuristic, the speed of a dry run depends less on the specifics of the build and should likely stay consistent as machine technology stays consistent.

Low Reliability, High Evolution: To determine design quality, check the color of the build for dark burn marks or a rainbow-like color, which can indicate weakened material properties.

High Reliability, Low Evolution: When performing the dry run, avoid going full speed so that you can visually confirm the spots being hit.

Figure 16 shows the combination of responses for heuristic evolution and its frequency of use. There was a significant negative correlation between the frequency of use of a heuristic and its evolution (Spearman’s ρ = −0.382, p < 0.001, and N = 120). This means the heuristics that participants tend to use more are perceived as changing less. As discussed in the previous correlation, the build order is likely used for most interactions on the EOS M280, as the heuristic is the result of machine-specific physics that occurs for every build. Consider the set of heuristics listed for low frequency and high evolution. It is possible that over/underbuilding occurs often, but the action chosen to address this is never consistent and is constantly being modified and improved upon. This decision could depend on several factors, such as surface quality requirements, machining availability, confidence in oneself to fix the issue mid-build, or how quickly one notices the issue.

High Frequency, Low Evolution: When setting the build order, build from bottom left to top right to minimize the impact of metal condensate.

Low Frequency, High Evolution: If the material is over/under the building, slow down/speed up the feed. If the material is over/under the building, change the work offset. If the material is over/under the building, and large unevenness of height occurs in the build, machine a few layers, then print afterward.

For the origin of heuristics, because up to four sources were chosen in some cases, the impact of the number of sources on reliability, evolution, and frequency of use was investigated. Results showed a significant negative relationship between reliability and the number of sources listed for its origin (ρ = −0.210, p = 0.020, and N = 122). This could be explained as if a heuristic is unreliable, participants are likely to search out other people and resources to help improve that heuristic. However, this is speculation and would require more study of heuristics for which a larger number of sources were listed.

The Kruskal–Wallis test was used to compare responses for evolution, reliability, and frequency of use across the other survey response attributes. This is similar to an ANOVA test but for nonparametric data. Therefore, we can judge whether heuristics that obtained certain attributes tend to have more or less reliability and evolution. The results found several significant differences in the data, listed below.

Heuristics with the following attributes were more likely to receive higher scores for evolution than those that did not receive these attributes:

• Considered risky, but saves time or other resources (H(1) = 9.671, P = .002);

• Applicable during Mid-Build (H(1) = 12.286, P < 0.001) or Quality Assessment (H(1) = 6.678, P = .010) stages;

• Originating from colleagues;

• Factors considered for implementation include input from other team members (H(1) = 8.613, P = .003), or experience with similar machines (H(1) = 4.026, P = .045).

Heuristics with the following attributes were more likely to receive lower scores for evolution than those that did not receive these attributes:

• Originating from industry or research standard (H(1) = 9.532, P = 0.002);

• Being a standard form of practice is a considered factor for implementation (H(1) = 20.225, P < .001);

• Listed as unsure of its origin (H(1) = 6.038, P = .014).

Heuristics with the following attributes were more likely to receive lower scores for reliability than those that did not receive these attributes:

• The context is easily recognized for application (H(1) = 5.515, P = .019).

• Applicable during Mid-Build stage of your process (H(1) = 3.829, P = .050).

Heuristics with the following attributes were more likely to receive higher scores for frequency of use than those that did not receive these attributes:

• Heuristics characterized as performed implicitly (H(1) = 7.558, P = .006).

• Heuristics characterized as valuable because they maintain safety (H(1) = 11.205, P = .001).

Heuristics with the following attributes were more likely to receive lower scores for frequency of use than those that did not receive these attributes:

• Literature was listed as an origin of the heuristic (H(1) = 6.250, P = .012).

• Listed as unsure of its origin (H(1) = 3.909, P = .048).

The correlations show that heuristics that were considered risky or applicable mid-build are constantly changing. This is possibly because participants are still trying to figure out the best way to attack those situations. As stated previously, decisions during the build are more so based on how the build is performing and is a case-by-case intuitive judgment. Therefore, it also makes sense that decisions mid-build were considered significantly less reliable as well. Heuristics originating from colleagues or dependent on team member input are also changing more than other heuristics. These changes may be due to the participant having to adjust input from others to work within their process. Initial advice may be from a colleague on experiences, but the participant’s experience may not line up to be exactly the same. Therefore, the heuristics passed on could require a trial and error process. The opposite seems to be true for heuristics originating from standards or implemented based on standards in place. It makes sense that decisions that appear to be standardized across the industry can be implemented without having to be modified frequently.

Reliability correlations showed that heuristics are used significantly less often when the participant believes the context to apply them is easy to recognize. It is possible that participants remember failures more than successes, and therefore, they can more easily recall situations where decisions have failed or been inconsistent in the past. The correlations with the frequency of use show that heuristics considered to be performed implicitly are used more often than others. This implies that participants do not typically explicitly think about their process in terms of going from one heuristic to the next. Safety-based heuristics are also used more often, showing that participants think a considerable amount in terms of maintaining the safety of themselves, their lab members, and the machines that are in use. Lastly, it was found that heuristics originating from academic literature are used significantly less often compared to other sources. It’s possible that academic literature is turned to for very specific scenarios and not general processes and therefore would only be necessary in a few cases. However, this is speculation and cannot be confirmed without additional studies.

Table 3 breaks down the heuristics extracted for each participant based on where they originated in the design journal questions. The visual moves from light to dark gray as the number of heuristics from a question increases. Heuristics were found successfully across questions asking for their process “before using the machine” and “while using the machine.” However, there were inconsistencies in how these questions were interpreted. On the EOS machine, the “build order” was discussed by one participant as “before using the machine,” and another participant “while using the machine.” Similar inconsistency was found in the Mazak participants when discussing the “dry run.” Three participants had heuristics discussing the dry run “while using the machine,” but one participant included this as “before using the machine.” Only one participant did not produce a heuristic “while using this machine.” Their response showed that they do nothing outside of troubleshooting besides ensuring the hopper is feeding the powder properly.

Participants overwhelmingly presented no new actions for when they would “not adhere” to the processes listed. The biggest explanations were that all steps listed were necessary and required for success. It was suggested that some steps could be relaxed if the part had been printed before or if the machine was already up and running by another user. Some information was simply reiterated. For example, one EOS participant P1 pointed out that sieving (the process of filtering out larger particles from excess powder recovered from a previous build) was only performed when necessary, but this was highlighted already in earlier portions of the journal. One participant added a note about cleaning the substrates. Only one participant presented information leading to a new heuristic. This Mazak user (P6) noted that they have to re-probe the work offset if the printed part is also going to be machined (the additive and machining heads are offset from each other).

For troubleshooting, the biggest factor on the EOS machine was to prevent recoater blade (the mechanism that spreads each layer of powder for fusion) crashes. Even when documenting outside of the “troubleshooting” journal question, a big purpose for many of the heuristics seen was to prevent these types of collisions. For the Mazak, obtaining proper powder flow was the target of many heuristics. One unique perspective (Mazak P7) was not focused on which actions to take to solve known issues, but rather on how they attack understanding what the issue is in the first place. For example, when the issue is not immediately clear, this participant focuses on narrowing down their problem to certain critical areas: CNC movement, feedstock, feed rate, and treatment, although these four areas were not defined in more detail during the interview.

Only one participant did not successfully write their personal rules of thumb in this section. They listed an example journal article reference containing design guidelines for laser bed powder fusion (LBPF) [45]. They did not specify which, if any, of those guidelines they used.

All participants successfully listed ways in which they inspect part quality. This includes knowing which methods to use to inspect quality: sometimes, it is a technology-based assessment (CMM, computed tomography (CT), etc.), and other times, it is a visual inspection.

The main point of Journal 2 was to ask participants about a recent, specific build, to help participants identify certain strategies they did not catch when completing Journal 1. In regard to this goal, not many new strategies were detected. Participants did not provide any significant changes to the strategies existing in Journal 1. For the EOS M280, all three participants explicitly stated no differences before or during use with the machine. For the Mazak machine, processes were mostly the same as well. Participant P7 relaxed some repetitive tasks based on their comfort level, and participant P8 reused a previous work offset and G-code, making some of their previous steps listed irrelevant. P4 noted they added a laser remelting strategy, but the participant considered the samples unusable and did not state any implications for using this strategy in the future. Lastly, P5 used a different maintenance process for this specific build because a new machine part had been introduced.

Only one new heuristic was added to the set from Journal 2. Participant P1 noted that a “soft” recoater brush might be necessary for future builds with delicate components. While the build did not produce a “failure,” some “struts” were damaged by the recoater action. A soft recoater brush, one that is more like bristles than a blade would allow the part to respond differently to brush contact.

Outside of the heuristic extraction examples presented in the methods section, several additional, noteworthy interview situations occurred, which are presented below without examples due to brevity. These include the following:

• The interviewer asked participants to be more direct regarding vague descriptors in their design journals such as “large” or “unusual.”

• The interviewer allowed participants to screen-share during the virtual interviews to provide a visual explanation or justification of some heuristics, such as the build order. Participants were not asked to do this, but rather it was done voluntarily by participants who felt that the most adequate justification or explanation would come through visuals.

• Participants provided the interviewer with the concept of learning heuristics by watching others, as shown by multiple Mazak users. These participants both acknowledged that they watch a fellow team member perform troubleshooting and picked things up this way, rather than a verbal or written exchange of information only.

The number of clarifications needed from the journals limited the amount of interview time discussing other aspects with participants, such as how the heuristics came to be. This led to more about context and less about mental processes to reach that decision. The amount of interview time needed per heuristic eliminated the ability to explore more of Journal 2. While few heuristics could be formed from Journal 2 alone, more time would have allowed the interviewer to navigate conversations and probe whether additional heuristics were possible to uncover. This is more of a limitation and tradeoff of the method chosen, as two hours is already a significant amount of interview time and data. A similar sentiment about interview time and method capabilities can be directed toward the key parameters that participants listed in the design journals. While additional information about parameters could have also been obtained through the survey, the survey was sufficiently long enough due to the quantity of heuristics and the number of questions devoted toward each heuristic. A comparison of process parameters across participants may show how each participant forms their perception of what is or is not valuable, although more information from the interview or survey method would be needed to relate this to heuristic value. Lastly, it would be valuable to spend interview or survey time discussing how users perceived the heuristics of other team members. This could help us evaluate overlap in the perception of decisions made. This also falls to the time and length limitations or interview/survey data and may be the basis of future work.

The interviewer’s lack of familiarity with terminology in interacting with the manufacturing technology may also be an influence on the results of the study. An interviewer with more experience may have saved time by not needing the participant to clarify certain terms, but it is also possible that being too familiar with the process leads to overlooking some necessary questions to uncover key elements of the heuristics. The interviewer in such a situation would be assuming knowledge that the interviewee is not verbalizing because of their own past experiences. An interviewer with more experience in manufacturing may also know what areas to probe during the discussion that an inexperienced person may not. However, this influence could also be negative, as it biases the conversation toward what the experienced interviewer considers important rather than what the participant values. In either case, it is possible that journal and interview assessments can lead to some differences in the overall repository of heuristics generated. This does not make the heuristics invalid but rather highlights the need for an iterative process or series of studies to converge toward a more complete and thorough heuristic set.

As discussed in Fig. 2, when participants were asked to confirm that they use the extracted heuristics, only 4 of 126 heuristics were listed as “somewhat disagree,” and no heuristics were listed as “strongly disagree.” It is unclear how these were invalid heuristics, as there was no follow-up discussion as part of the research study. It can be noted that three heuristics were taken from journal responses that required additional information to develop the heuristic, similar to “Example Three” in the described heuristic extraction process. An example is shown below:

Journal Question #9: Describe how you determine if your part has been built successfully. What are the things you look for when determining quality?

Participant Response: “…Depending on the requirements, this could require inspection via hand tools or other metrological techniques (XCT CMM, surface metrology, etc.).”

Interviewer: “You listed ways to inspect part quality, which was CT, CMM, and surface metrology. Are there any of these that you use more than others, or do use these at all for your work? Can you give a rundown of when you would use one over another?”

Participant: “Yeah. So, I've used all of those in my research. And each has their own benefits…The last one is computed tomography… CT, however, is an extremely complicated measurement procedure. And while it is able to give some pretty awesome results, it is not technically like a traceable measurement technique. So, like any dimensional measurements that you take on a CT have to be taken with, sort of like a grain of salt. That, like we're not actually sure how uncertain we are in this measurement. But that being said, you can still do a number of analyses with it, which are mostly at this point comparative, like you're not able to take like an absolute measurement of diameter or something like that. But what we've used before is, like you know, comparison of like this process to this process. This part to this part I'm looking at comparisons…”

Extracted Heuristic: To assess quality through point comparison relative to other parts or processes, use CT technology.

As discussed in the background section, Haselton presents three ways in which humans rationally adapt for survival: heuristics (saving time and resources in exchange for a potentially sub-optimal outcome), error management (acting towards less costly error—false positives are less costly than false negatives), and experimental artifacts (a product of poor research design which produces unnatural or unusual environments) [11,12]. It is possible to view the results of this study in terms of error management—how participants may have perceived the costs of using their heuristics as far less than the costs of not adhering to them. This section presents several extracted heuristics in many different contexts. The objective is to show the diverse ways in which participants give up time or resources if it means ensuring part quality or machine safety. Each example highlights how participants were willing to sacrifice some costs to preserve their respective builds and machines. This willingness to operate as if an error would occur otherwise is the basis for each “false alarm” case or false-negative.

The previous presentation of results showed that only six heuristics in total were characterized as risky, and many of these related to participants making intuitive judgments in the middle of a build: “If the build makes noise due to significant overbuilding, manually slow down the feed rate and deposit more material in lower areas to even out the part.” In this situation, the participant must decide whether the costs associated with unnecessarily stopping and manually controlling the build (false-positive) are less than the costs of letting the build continue and resulting in undesirable part quality (false-negative). The justification for stopping the build would be that more false alarms are better than more misses.

Figure 8 shows the attributes that participants attached to heuristics as reasons why the heuristic maximized the value of their process. Near the bottom of the list of value-producing attributes was improved efficiency in terms of time (24% of heuristics) and material/resources (20%). At the top of the list, the attributes attached to the most heuristics were to achieve desired part dimensions (56%), achieve other aspects of part quality (50%), and prevent machine failure (48%). From an error management perspective, a false-positive would be to spend extra time/material to ensure there is no detriment to the part or machine quality, although the resulting build session shows that the extra time/resources were not necessary. A false-negative would be to save time/material and have a situation occur where the machine or part quality diminishes. The data from Fig. 9 imply that participants likely consider the costs of the false-positive to be far less than the costs of the false-negative. In other words, their efficiency is seen through preventing subpar machine or part quality, rather than preserving their own time and resources.

Statistical correlations showed that heuristics characterized as maintaining safety were applied significantly more than other heuristics. From an error management perspective, participants may believe that the costs of implementing a safety-based heuristic without it being needed (false-positive) are less than the cost of not implementing the heuristic and safety being compromised (false-negative). In the false-positive, additional time and resources may be used, but this cost does not compare to costs that may threaten the health of the machine or its users.

It is possible to see some error management perspectives in how the heuristics have been presented. For example, consider the heuristic: “When setting the build order, build from bottom left to top right to minimize the impact of metal condensate.” The action in this heuristic is to set the build order from bottom left to top right. However, the justification of this heuristic is to minimize the impact of metal condensate on the part. The participant may understand that taking the extra time to set up a build in this order, no matter how much condensate may actually impact the part, is much less of a cost than producing a separate build order that produces an unreliable part due to contamination.

The interviews contained some conversations in which participants admitted some level of being risk averse. For example, participant P2 agreed that they may have avoided the use of supports when they were not as familiar with them. Instead, they would default to modifying the orientation or changing the part entirely. In those situations, the costs associated with modifying the part or orientation may have been less than the costs of choosing the wrong support and having a failed build. Participant P7 admitted to staying closer to the machine when they were afraid of collisions because of past mistakes. In this situation, the cost of staying near the machine and having a successful build (false-positive) is less than the cost of leaving the machine and being unable to intervene when necessary (false-negative).

This study provides the field of manufacturing with a framework for obtaining and characterizing heuristics, which is beneficial as new technology, such as hybrid manufacturing, continues to grow and evolve. The results show statistically significant correlations between heuristic reliability, evolution, and frequency of use. This validates prior work in heuristics and adds these correlations to the field of advanced manufacturing for both designers and machine operators [4]. The survey results show which heuristic attributes statistically significantly impact the perception of heuristics as reliable, evolving, or frequently implemented into one’s process. Lastly, a new perspective of heuristics in advanced manufacturing was shown in which participants’ progress toward heuristics results in the least costly errors.

For the heuristics collected, translation outside of this study may rely on machine quality and their current technology levels, the experience of the user obtaining these heuristics, or the objectives for the use of their respective manufacturing machine. As expected, case study research generally comes with limited application of results beyond the case being studied. However, these results serve as a starting point for hypothesizing heuristic use across other populations of designers, which can be tested by comparing additional case studies or creating new controlled experiments to test our findings. These findings can be used in future work toward the original research questions:

How should the methodology for extracting heuristics be improved such that we may assess the value a heuristic brings to the design process?

Results showed that certain origins of heuristics correlate with heuristic evolution, and a higher number of sources led to decreased heuristic reliability. The methodology should include an iterative process that includes additional interviews after the survey phase. This will allow more understanding of what information was taken from each source, and the lessons learned from participants using that information. The iterative process may account for addressing discrepancies where participants did not fully agree with the final set of heuristics extracted.

What aspects of heuristics and design environments should be considered during documentation of heuristics in a repository?

Results showed that some characteristics of heuristics (such as origin, applicable process stages, user perceived characteristics, and factors for implementation) imply more evolution over time. This can impact the rate at which particular heuristics are or should be reassessed and updated in a repository. Staggering the rate at which certain sets of heuristics are updated could improve the efficiency of maintaining heuristic knowledge.

How might heuristics be characterized and classified to understand their impact on design processes?

Heuristics should be characterized not only as context-action, but by their sources of origin, applicable process stages, and characterizing descriptors based on perception from previous users. These factors were found to have an influence on which heuristics may be more/less reliable or applicable more/less often to the process. Heuristics in context-action form should be assessed in comparison to heuristics reframed in an error management-based form, which may help determine which type of framing resonates more with users.

The authors would like to thank Dr. Tom Kurfess and Dr. Chris Saldana for their guidance throughout this research. This work was supported by the National Science Foundation under Award CMMI-1,846,048. This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05—00OR22725 with the US Department of Energy (DOE). The publisher acknowledges the US government license to provide public access under the DOE Public Access Plan.²

There are no conflicts of interest.

The data sets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.