A Computational Model of Human Proficiency in Engineering Configuration Design

Teams are the cornerstone of engineering organizations; however, much is unknown about how best to construct them. Several frameworks have been used to describe the team composition and creation process [1]. It is not disputed that the composition of individual team member attributes plays an important role in a team’s performance. Still, there are inconsistent results on whether an attribute’s minimum level, maximum level, variance, or average level is the best predictor of team success [2]. The relative contribution framework is one such framework and focuses on how individual team members affect team outcomes relative to the other members of the team [3]. A previous human subject study demonstrated how the proficiency of individual designers contributes to overall team performance in a configuration design problem [4]. It was found that the most proficient member had the most significant effect on team performance metrics, positive or negative. In short, the maximum level of proficiency on the team is the best predictor of team success. This means that increasing the proficiency of the most proficient member of a team has a larger effect than increasing the proficiency of any other team member by the same amount. While this result is insightful, the limitations of a human subject study constrain the conclusions that can be drawn. Because of the high time cost of a human subject study, a computational model of proficiency in engineering configuration design would be beneficial and complement the findings from human subject studies. With a validated computational model, the challenges of limited data and isolation of desired attributes of team members can be mitigated.

A single computational agent represents each designer in a multi-agent model. Autonomous agents are modeled such that they perceive their environment, exhibit goal-directed behavior, and are capable of interacting with other agents [5]. Multi-agent models have been used to model various aspects of design, including design organizations [6,7], design negotiation [8], optimal team structure [9], navigating disrupted design objectives [10], social learning in teams [11], and multidisciplinary design [12]. Recently, heterogeneous agents have been utilized to evaluate how varying characteristics of an individual agent affect team outcomes in multi-agent models of engineering design teams [13–15]. While many other multi-agent models have been used to solve various engineering problems, they are designed to either solve the problem or propose new design methods, not to model and study engineering design teams and their problem-solving strategies [16–18].

The objective of this work is to model proficiency in engineering configuration design. Configuration design is a class of design tasks where the artifact being designed is constructed from a set of predefined components and obeys a set of constraints [19,20]. A validated model can draw conclusions about how different proficiency designers should be utilized to construct design teams more effectively. Proficiency is a designer’s ability to deal with a specific range of problem [21]. It is related to a person’s general aptitude and the complexity of the task but is most strongly correlated with the person’s experience with the task [22]. To model an attribute, there must be behavior differences for varying levels of that attribute. In the previous human subject study, high-proficiency designers moved quickly from design initialization to fine-tuning and optimization and spent less time building the topology of their design. Meanwhile, low-proficiency designers spent significantly more time developing a topology of their design before moving onto the optimization phase of the design process [4]. This finding fits the existing literature on expert versus novice designer behavior, which can translate to high and low proficiency. A well-established element of design expertise is a solution-focused approach. Experts move to a problem solution faster than less experienced designers and spend less time exploring the problem domain [23–25]. The other difference in behavior for various proficiency designers is that different proficiency designers utilize distinctive move selection heuristics [4]. These heuristics were explored using extracted hidden Markov models (HMMs), which are shown to model the sequential design process heuristics effectively [26]. These two findings can be modeled to create cognitively inspired computational agents that mimic the behavior of different proficiency designers and consequently mimic the performance of different proficiency designers.

In summary, this paper aims to introduce and validate a cognitively inspired computational model of proficiency in engineering configuration design. This model can serve as a platform for future experimentation to better understand individual team member proficiency and its effect on team performance. The model implementations described in this work are available to the public for educational and academic research purposes.²

This work introduces the Proficient Simulated Annealing Design Agents (PSADA) model. The inspiration for this model is the cognitively inspired simulated annealing teams (CISAT) model introduced by McComb et al. [27]. Like CISAT, each PSADA agent uses the simulated annealing stochastic optimization method to dictate their design process [28]. Simulated annealing describes significant aspects of human problem-solving behavior [29] and has been used to computationally model human behavior on mechanical design problems [27,29]. PSADA differs from CISAT by modeling agents heterogeneously, with varying proficiency levels. To accomplish this goal, PSADA models agents using characteristics that focus on individual designer decisions in the design process. These characteristics are modeled after the behavior of individual designers identified from a previous human subject study. Specifically, two characteristics of individual designers are explicitly modeled to mimic those behavior differences for various proficiency designers. They are:

1. Design space search strategy—Describes a designer’s tendency to utilize breadth or depth search strategies based on proficiency [4,24].

2. Move selection heuristics—Designers utilize past relevant experience to guide sequential design decisions [4,26,30].

Both characteristics and how they are modeled are discussed in detail in Secs. 2.1 and 2.2.

The different proficiency PSADA agents are validated against their equivalently proficient human designers on two engineering configuration design problems. After validating the different proficiency PSADA agents, they are placed into teams and interact in the same method as the human subject study. This team structure for configuration design problems mimics the team structure of sub-teams within a larger design team. For example, when a company designs a car, several sub-teams work on specific components, like the suspension, chassis, and powertrain. Each sub-team member can collaborate with their sub-team teammates but could realistically develop a solution independently.

Figure 1 summarizes the structure of the PSADA model for both an individual agent and a team of agents. In the PSADA model, each agent can work on the entire problem. Each agent iteratively works to create a design, shares their solution with teammates at probabilistic, self-determined moments, and probabilistically chooses whether to adopt their teammates’ designs when they are shared. This structure allows for individual agents to be modeled independently from their teammates; thus, the characteristics of individual agents and their effect on team performance can be analyzed. Agents complete all actions and decisions independently, and therefore, the model is easily scaled to include different numbers of agents on a team. Section 2.3 discusses the team-based characteristics in detail.

Analogous to high-proficiency designers, expert designers often move more quickly to a solution path [24]. Analogous to low-proficiency designers, novice designers often use a “trial-and-error” approach [31]. It is posited that these differences in design space search strategy are due to experience and understanding of the entire design space. High-proficiency designers can quickly make conjectures about solution paths. In contrast, low-proficiency designers must take time to attempt solution paths and make decisions only after they gain feedback from their attempts [4]. This key insight is one way in which proficiency is modeled in PSADA.

For this equation, q is the quality of the previous iteration, q′ is the quality of the new iteration, T is the temperature parameter, and it assumed that q′ > q. The temperature parameter in simulated annealing decreases based on a cooling schedule throughout the search process. The initial temperature for the algorithm depends on the objective function used.

In any design process, designers use heuristics to guide them in solving the design challenge [33]. These heuristics are learned from past relevant experience with the task at hand. The move selection heuristics can be extracted using HMMs for a configuration design task [4,26]. An HMM represents a first-order Markov process with hidden states [34]. An HMM is represented by a transition matrix, T, and an emission matrix, E. After each step, the T matrix shows the probability of the next step being in each state. At each step, the E matrix shows the probability distribution for taking each action given the current state at that step. Experimental data are required to train an HMM for each level of proficiency. A PSADA agent will probabilistically select their action at each iteration, given their proficiency and current state. After each iteration, a PSADA agent will probabilistically transition to the next state, given their proficiency and current state.

A designer’s proficiency at a design task is most strongly related to their experience with it [22]. In PSADA, each level of proficiency requires a unique HMM trained on past human behavior for that task. Therefore, the task-specific element of proficiency is captured by these move selection heuristics in the PSADA model.

The PSADA model adopts team-focused characteristics from previous models of design teams that contribute to how agents solve problems as part of a team [27]. The modeling of each characteristic in PSADA is designed to focus on individual agent decision-making instead of team decision-making as in other models. Specifically, the communication in the PSADA model is designed so that agents independently determine when to communicate their solutions with their teammates, and their teammates assess other designs only when they are shared. The team-specific characteristics that are used in this model are:

1. Organic interaction timing—Interaction occurs at irregular and self-determined intervals [35,36].

2. Quality bias reduction—Individuals in a team develop multiple solution concepts to avoid premature convergence [37].

3. Self-bias—Designers are biased toward their own solutions [38].

In Eq. (3), P_share is the probability of sharing the new solution, q′ is the quality of the new solution, and q is the quality of the previous solution.

In Eq. (4), q_s is the objective function value for the shared solution and q_c is the current objective function value for the agent. In Eq. (5), P_switch is the probability of adopting the new design, and β₀ and β₁ are the coefficient estimates.

Finally, designers have been found to favor their own designs, even when it is not the best option available to them [38]. Because PSADA models interaction as a binary choice using logistic regression, the probability of switching designs can determine if designers in the original human subject study favored their own designs and, therefore, whether the PSADA model incorporates self-bias.

Proficiency is inherently task-specific. The PSADA model is tested on two different configuration design tasks to ensure that the model is generalizable for modeling proficiency across configuration design problems. Table 1 summarizes the parameters that are used for each problem.

The insights that inspired the PSADA model are from a previous human subject study conducted by Brownell et al., in which human designers (n = 78) were tasked with designing various truss structures that met specific constraints [4]. Previous studies have shown that truss design is a representative configuration design task, and the results can translate to other configuration design problems [26,39].

To design a truss, designers utilize a sequential design process of building a truss structure to support loads given anchor nodes. Figure 2 shows the truss design problem used in this PSADA work. This problem represents a familiar yet non-trivial problem to mechanical engineers. Any two-dimensional truss problem can be used in the PSADA model.

In Eq. (6), q is the quality of the solution, m_t is the target mass based on the specific truss problem, m_a is the mass of the current truss, FOS_t is the target factor of safety, which is always 1.25, and FOS_a is the lowest factor of safety for any single truss member. This objective function is the truss’s strength-to-weight ratio (SWR).

HyForm is a platform that was developed to study hybrid teams. This platform includes an unmanned aerial vehicle (UAV) design interface, shown in Fig. 3. UAV design in the HyForm platform is a configuration design problem, where the UAVs are comprised of motor–rotor pairs, battery structures, and foils connected in a grid. Designers can construct a UAV and simulate its validity and performance [40]. Table 3 shows the possible actions, along with a brief description of their UAV PSADA implementation.

An experiment by Zhang et al. published data for individual designers (n = 20) building drones to meet specific objectives [41]. These data on individual designer behavior and performance are used to extract move selection heuristics via HMMs for different proficiency drone designers to build PSADA model agents for UAV design. The first set of objectives for the human subject study was to “construct a fleet of UAVs that [are]: (1) Capable of flying a total distance of 20 miles and (2) capable of flying a total distance of 50 miles.” The objective function for the UAV design problem is, therefore, simply the distance that the UAV can travel.

Two significant groupings are found in this dataset of human designer performance. Therefore, no medium proficiency designers were categorized in this dataset. Human designers are categorized into high- and low-proficiency designers. HMMs are derived from the action sequences of these high- and low-proficiency designers.

The results aim to validate the model of engineering proficiency proposed in Sec. 2. This is done by comparing results from the computational model with the previously conducted human subject studies. The only parameters that change across design tasks or for different levels of proficiency are those that are summarized in Table 1.

Proficiency is inherently an individual characteristic. Therefore, the proficiency of individual agents must be validated before they are put into teams. The human subjects designed trusses and UAVs in the same scenarios as the agents, which enables a comparison of PSADA and human designers. As discussed in Secs. 3.1 and 3.2, the human data from the truss design task had three distinct proficiency levels, and the human data from the UAV design task had two distinct proficiency levels. Therefore, the PSADA model implementation has high-, medium-, and low-proficiency agents for the truss design task and high- and low-proficiency agents for the UAV design task. One hundred individual PSADA model agents of each proficiency level are instantiated, and their results are analyzed.

As a reminder, the PSADA model aims to model characteristics of designers with different proficiency levels to create agents that mimic human designer behavior and performance. In other words, the agents’ behavior should match that of human designers with the expectation that if the agents behave like various proficiency designers, they will perform like various proficiency designers.

Therefore, the behavior of PSADA model agents is compared to human designers on the two design problems first. Figures 4 and 5 show the action frequency of the different proficiency agents and the different proficiency designers in the truss and UAV design problems, respectively. The different proficiency human designers exhibit different action frequencies, and move selection heuristics that are modeled in PSADA are able to capture these differences for both the truss and UAV design problems. By visual inspection, the different proficiency agents select actions with similar probabilities to the corresponding proficiency human designers. In the truss design problem, the KL divergence for each proficiency pairing (high–high KL: 0.0128, medium–medium KL: 0.0180, low–low KL: 0.0157) confirms that the distributions are closer than each proficiency non-pair (high agent–medium human KL: 0.1543, medium agent–high human KL: 0.0625, low agent–medium human KL: 0.0633). For succinctness, the KL divergence for the furthest apart distributions is not included. The same is true for the UAV design problem (high–high KL: 0.0050, high agent–low human KL: 0.0739, low–low KL: 0.0073, low agent–high human KL: 0.0646).

The design space search strategy is the other behavior that is explicitly modeled in PSADA. In the original human subject study, it was discovered that low-proficiency designers spent longer exploring the design space before selecting a design topology and refining it. This was uncovered by analyzing the designs being created, including the number of topology elements in the designs. For truss design, the topology elements are the nodes and the members of the truss. Figure 6 replicates this analysis for the PSADA model agents and compares the results to the human designers. Because the design space search differs for different design problems, the equivalent analysis does not hold the same meaning for the UAV design problem. The topology of a UAV interacts differently for a UAV design problem. The number of topology elements for low- and high-proficiency designers were compared and 300 two-sample t-tests resulted in zero significant differences for this analysis. Thus, the comparison between human and agent designers was not conducted.

Figure 6(a) shows that the truss designs that are created by the different proficiency agents replicate the results from the different proficiency human designers (Fig. 6(b)). In both cases, all designers add topology elements to their design until approximately action 20, which is the required number of actions required to create the most common truss topology for the design scenario that is used. After building the initial truss topology, higher proficiency human designers and PSADA model agents do not add as many topology elements during the remainder of the design session. Low-proficiency human and agent designers, however, continue to add topology elements to their truss designs.

The different proficiency PSADA model agents were compared to their respective proficiency level human designers using two-sample t-tests. Only the iterations after the 20th action were compared. In total 204 two-sample t-tests were conducted and thus, the Holm–Bonferroni correction is used to set the significance levels. Of the 204 tests, only 11 of them yielded significant results. The tests for the high proficiency designers in action numbers 21–27 (p < 0.00026) account for seven of the significant differences. After this initial difference, the high-proficiency human and agent designed trusses do not have a statistically different number of topology elements. The final four iterations (p < 0.00091) for the low-proficiency designers yielded statistically significant results. In general, these results demonstrate that the agents are not designing trusses that are significantly different from the human designers and that all proficiency level agents are exhibiting the appropriate search behavior.

Upon validating the behavior, the averaged performances of the different proficiency human designers versus the averaged performance of the different proficiency PSADA model agents are compared and shown in Figs. 7 and 8 for both design problems. For the truss problem in Fig. 7, all agents and human designers have a mean quality of their first stable and statically determinant truss slightly below 0.3. The final values of each level of proficiency are consistent between the two as well. Beyond the initial and final quality, the trends for all three proficiency levels are reproduced by the PSADA model, with some minor exceptions. For each iteration, a two-sample t-test is conducted between the mean truss quality for the high, medium, and low proficiency PSADA agents and human designer pairings. One hundred and fifty-four significance tests are conducted, and the Bonferroni correction is used to set the significance level. For the high-proficiency and medium-proficiency designer and agent pairings, the tests failed to find a significant difference between the means of the two populations at any iteration. For the low-proficiency designer and agent pairings, only iterations 9, 12, and 15 resulted in statistically significant differences.

For the UAV design problem in Fig. 8, all human designers and agents are given a starting basic UAV design, and therefore all had a minimum best UAV range of ∼10. Early in the design session, the average design quality is not significantly different between the high- and low-proficiency human designers or agents when using a two-sample t-test at a p < 0.01 significance level. A stricter significance level was used due to the high number of significance tests being conducted. Only after the 224th action for the human designers (p = 0.00180) and the 228th action for the agents (p = 0.00979) do the high-proficiency designers have significantly better designs on average than their lower-proficiency counterparts. The average design quality is never significantly different when comparing the low-proficiency human designers and agents. However, the difference between the design quality for high-proficiency designers and agents is significant late in the design session. The human designers achieved a significantly better UAV quality after the 237th iteration. Despite the difference between high-proficiency agents and human designers, there is still a significant difference between low- and high-proficiency agents.

Although many of the high-proficiency agents achieve high ranges for their UAVs, which is defined as a range > 50 (the high proficiency human designer range criteria), not all are able to achieve this. The UAV design problem has a complex design space with large areas of unstable UAVs with an objective function value of zero. Some agents get trapped and spend long periods in zero-quality design space areas, limiting their ability to produce high-quality designs. The model is built first to mimic the human designer’s behavior and then to determine whether the behavioral differences modeled between different proficiency designers can account for the differences in performance. Ultimately, the PSADA agents are not able to escape these zero-quality regions in the same way as the human designers for this single proficiency level.

Another limiting factor for the UAV design problem agents from mimicking the high-proficiency human designer performance is the limited number of high-proficiency human designers. While there are enough designers to optimize HMMs for high- and low-proficiency agents, the HMMs would improve with additional human subjects. Because the dataset is from published data instead of the author’s work, additional data are unavailable. Despite these issues, the PSADA model can still produce high-proficiency agents with significantly better-quality UAVs than their low-proficiency counterparts.

While Figs. 4–8 demonstrate the validity of the PSADA model in two different design scenarios, it is also important to demonstrate that the individual characteristics that are modeling proficiency—design space search strategy and move selection heuristics—contribute to these results. To test this, each characteristic is “turned off,” one at a time, so the modified PSADA model can run without the characteristic to show how the results change. The results of this experiment with the truss design problem are in Fig. 9.

The design space search strategy characteristic determines the cooling constant, c_p in Eq. (2), for each level of proficiency. To “turn off” this characteristic, each proficiency level is given the same cooling constant. In the test, each is given a cooling constant of 0.95. The rest of the PSADA model remains unchanged (i.e., the different proficiency agents still had unique HHMs) during this test, and 100 instances of the modified model are run with individual agents. Figure 9(a) shows the results of this test.

The move selection heuristics use HMMs to probabilistically choose an action at every iteration. Each level of proficiency has a different HMM derived from human data. To “turn off” move selection heuristics, each level of proficiency selects a random action at every iteration instead of choosing an action based on the HMMs. The rest of the model remains unchanged (i.e., each proficiency level had a unique c_p) during this test, and again 100 instances of the modified model are run with individual agents. Figure 9(b) shows the results of this test.

As Fig. 9 shows that both characteristics significantly impact the design quality progression for all proficiency levels when compared to Fig. 7(b). When both characteristics are used, PSADA agents closely follow human designers’ behavior and performance trends. When one of the characteristics is not modeled, the performance differences are not fully captured. These results demonstrate that the characteristics modeled in PSADA capture the behavioral and performance differences between the different proficiency human designers in configuration design tasks.

With three different proficiency agents modeled in PSADA, 10 different team configurations are possible for a team of three agents, one with three low proficiency agents, one with two low and one medium proficiency agents, one with two low and one high proficiency agents, etc. Each condition is run for 250 instances, and each team acts for 70 iterations. The results are shown in Fig. 10. The coefficient estimates for β₁ (the effect size of the lowest proficiency team member), β₂ (the effect size of the medium proficiency team member), and β₃ (the effect size of the highest proficiency team member), as shown in Fig. 10(b) show that improving any member of the design team will result in a positive effect on team performance. This result fits the logic that better team members will lead to better team performance. What is not as predictable is that raising the proficiency of the most proficient member has the largest effect on team performance. Therefore, the computational study with PSADA agents confirms the results from the previous human subject study that improving the highest proficiency member of a configuration design team will have the largest positive effect on team performance. This also validates the team characteristics of the PSADA model. With Welch’s t-test, this result is statistically significant when comparing β₁ (lowest proficiency team member) and β₃ (highest proficiency team member) (p = .0108). However, the differences in effect size are not statistically significant when comparing β₂ (medium proficiency team member) and β₃ (highest proficiency team member) (p = .7793) or comparing β₁ (lowest proficiency team member) and β₂ (medium proficiency team member) (p = .0757). The general trend shows that the more proficient a team member is relative to their teammates, the larger their effect on team performance.

A computational model of configuration design proficiency is introduced in this work. The Proficient Simulated Annealing Design Agents model is shown to accurately model the behavior of human designers in two different configuration design problems. Additionally, the differences in behavior lead to significantly different performance levels, mimicking human performance with just one exception. In a truss design problem, three proficiency levels are validated against human subjects from a previously conducted human subject experiment. In a UAV design problem, two proficiency levels are modeled and compared to human subjects’ behavior and performance. The behaviors of action selection and design space search are accurately modeled in both problems. The performance of all three proficiency levels in a truss design problem is accurately matched. The performance of the low-proficiency designers accurately matches the UAV design problem. The high-proficiency agents perform significantly better than the low-proficiency agents but do not mimic the performance of their human-designer counterparts. The reason for this difference in performance is discussed.

Upon validation, PSADA agents can be placed into teams to collaboratively design configurations with the same effect as their human counterparts. For the truss design, it is confirmed that the highest proficiency team member has the largest effect on team performance. This shows the PSADA model’s ability to computationally model certain engineering design teams. In this introduction to the PSADA model, individual proficiency in configuration design problems is studied. However, now that the model has been validated to mimic human designers of different proficiency levels and design teams, PSADA can be utilized in future experiments to test other characteristics, other engineering design tasks, other team structures, and engineering design organizations that distribute designers to multiple teams simultaneously.

PSADA allows for an in-depth analysis of designer proficiency’s role in configuration design teams. With this model, it can be confirmed that the previously discovered behavior differences between various proficiency designers account for a large portion of the variance between different proficiency human designers in engineering configuration design. The PSADA model does not currently model other characteristics that interact with proficiency. Personality, operational learning, and leadership could affect how low-proficiency designers learn during a design session to become higher-proficiency designers and how they use their knowledge and expertise to influence lower-proficiency teammates positively. Incorporating new cognitive and behavioral characteristics into the PSADA structure is possible and is a fruitful area for future work. Additionally, the PSADA model requires human data for the move selection heuristics. Without a previously conducted human subject study to train HHMs for different proficiency agents, applying the PSADA model to new design tasks would not be possible.

It is impractical to assume that every designer will be of the highest proficiency and that understanding how best to utilize people of different proficiencies can lead to better outcomes. This work shows that the PSADA model is a foundation for a deeper understanding of the role that individual team member proficiency plays in engineering design teams, yielding novel insights into this area.

This material is based on work supported by the United States Air Force Office of Scientific Research (ASFOR) through Grant No. FA9550-18-1-0088. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the sponsors.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.