A Computational Study of Design Team Robustness Through the Lens of Cognitive Style

High-performance design teams must achieve a level of robustness—the ability to perform well and consistently across a variety of complex design tasks. In practice, robustness is often achieved through careful management of the design process. The literature emphasizes the importance of team management to reduce failures [1,2], and team management is often regarded as the backbone structure of any organization [3]. However, there exists a potential to carefully compose and structure teams so that they are more likely to be inherently robust. Cognitive style, the preference for different levels of structure in problem-solving [4], presents a significant opportunity to form robust teams based on the individual differences of team members. In this work, we examine the robustness of teams with different structures and cognitive style compositions through a computational simulation study.

Cognitive style, particularly as defined in Kirton’s Adaption-Innovation (A-I) theory, directly addresses dimensions of problem-solving that are relevant for engineering and design [4]. Cognitive style refers to the natural and preferred way of individual processes and organizes information [5] which in turn influences the problem-solving process [6]. The A-I theory is thus concerned with how people solve problems. This is often contrasted with Kirton’s definition of cognitive level, which refers to an individual’s capacity for problem solving and creative behavior and is often measured with intelligence tests [7]. The cognitive level is independent of cognitive style and representative of the potential capacity to solve problems. While varying cognitive levels are necessary for large, complex problems [8], the correct cognitive level is often clearly related to the task at hand. Differences in cognitive style, however, produce distinct patterns of behavior that interact when people solve problems collaboratively. Both cognitive style and cognitive level influence one’s ability to solve problems. Therefore, cognitive style from an A-I theory perspective is a potential avenue of which to study robustness with respect to team composition and structure. Differences in cognitive style produce distinct patterns of behavior which are important, when people solve problems collaboratively, independent of the cognitive level of the individual.

To measure the cognitive style in the problem-solving context, Kirton developed the Adaption-Innovation Inventory (KAI). The KAI places one’s cognitive style on a numerical bi-polar continuum, ranging from highly adaptive to highly innovative, with a theoretical range of 32–160 [9]. More “adaptive” problem-solvers prefer to work with more structure, prefer known solutions, and make incremental changes while seeking to maintain group cohesion [8]. On the other end of the spectrum, more “innovative” problem-solvers are more tolerant of a loosely-guided structure, cutting across paradigms and the existing structure to solve problems “differently”, with less concern for group consensus [8]. Kirton referred to innovators as those who have a tendency to “overhaul the entire work process”, approaching tasks from unexpected angles and pursuing radically different solutions with less regard to existing structures [10,11]. This continuum is rooted in the axiom that wanting “to do things better” (i.e., the adaptive end of the spectrum) and wanting “to do things differently” (the innovative end) are equally creative, although different styles may be desirable depending on the situation [12]. Through multiple validation studies, Kirton also developed three sub-scores that are based on the sub-factors of cognitive style. The sufficiency of originality (SO) sub-factor highlights the differences between individuals and their preferred ways of generating and offering ideas. The efficiency (E) sub-factor reflects an individual’s preferred methods for gathering and organizing ideas. The rule/group conformity sub-factor (R/G) reflects differences in the ways individuals manage the personal and impersonal structures in which their problem solving occurs. Earlier work has leveraged the A-I theory to determine how the KAI score affects individuals’ ability to problem solve when given adaptively or innovatively framed design problems [13].

Kirton’s theory has been routinely applied in the earlier literature to examine the cognitive processes of collaborative teams. KAI has been used to determine the cognitive styles of engineering and management students in project teams, which showed correlations between positive and negative aspects of project team experiences [14]. When studying the relationship between cognitive style and team interactions during concept generation, the results suggest that more innovative teams tend to have interactions with a higher degree of integration between topics [15]. Another study examining team success and cognitive diversity using the KAI measure to examine team cognitive styles found that teams had higher levels of success when tasks were coordinated with team members’ KAI scores (e.g., a more adaptive task with a more adaptive sub-team) [16]. It was also found that homogeneous teams were successful due to enhanced inter-team communication, whereas heterogeneous teams fell as a result of unresolved cognitive gaps [16]. In terms of creative fluency, another researcher found that heterogeneous teams outperformed homogeneous teams in terms of creative output [17]. These studies establish KAI as a particularly useful framework for assessing the impact of cognitive style on performance within design teams.

Two important dimensions for describing a team are composition and structure. Team composition is often viewed in terms of homogeneity or heterogeneity. While there is recognition that heterogeneity in teams benefits with creativity in problem-solving, homogeneous teams may be better at the exploitation of existing knowledge [18]. The team structure is often examined through the degree of hierarchy within the team. Teams often form implicit hierarchal structures (e.g., sub-teams) once they reach a certain size [19]. Subgrouping has been shown to enhance creativity within virtual and collocated design teams [20]. Furthermore, some studies indicate that teams achieve better performance with frequent interaction and communication across sub-teams [21,22].

As teams become more interdisciplinary and develop more context structures, selecting the right members becomes more critical [23]. However, the research on optimal team formation strategies is limited. Studies on design teams in vivo are often expensive and applicable only to specific contexts within their scope [24]. Therefore, this work utilizes agent-based modeling (ABM), a type of computational simulation, instead of conducting traditional human subjects studies. This makes it possible to rapidly vary team composition and structure to examine robustness while carefully controlling specific variables, which is challenging in human subjects studies. Agent-based modeling to simulate teamwork has been undertaken frequently in the literature [25–28] and used to understand human systems in the design [29]. One study, in particular, used ABM to examine how team member selection and diversity affect team performance in various economic conditions [30]. Using homogeneous, heterogeneous, and interdependence-based strategies to form project teams, the model explored the relative effects of different team member orientations on firm performance or team profit. This study showed that while the difference in profit earned from homogeneous and heterogeneous teams is not significant, heterogeneous teams can enhance the overall firm performance in certain conditions. Recently, some studies have also used ABM to model the social and emotional aspects of team problem solving [31], while others have used ABM to model the effects of stress and motivation on team performance [32].

In this work, we study simulated cognitive styles informed by the KAI measure. Adaption-Innovation theory has been directly used in the KAI agent-based organizational optimization model (KABOOM) to simulate the cognitive style in problem-solving teams. KABOOM simulates team problem-solving by executing a modified multi-agent simulated annealing algorithm [33]. In KABOOM, a team of agents begins a stochastic search of the solution space and gradually transitions to a downhill search to refine a solution. Simulated annealing has been validated earlier as an effective model of human problem-solving behavior [34–36]. The three main modifications from a basic simulated annealing algorithm include agents possessing unique cognitive styles to modify their exploration of a solution space, teams of agents who specialize in decomposing a problem into sub-problems, and agents communicating to share solutions in pair-wise and team-wise meetings [11]. Autonomous agents with various cognitive styles interact to solve a diverse set of problems and maximize an objective function, indicating performance in KABOOM. KABOOM simulates cognitive style by assigning an overall KAI score and related sub-scores to an agent. The overall KAI score affects how agents perceive the solution quality and their exploration of the solution space, where more adaptive agents take smaller steps throughout the simulation and more innovative agents take larger steps. The KAI sub-scores for each agent further represent the dimensions of cognitive style and model more specific aspects of human problem-solving behavior and solution generation. While KABOOM assumes that agents can evaluate the quality of their solutions using the objective function, their perception of the solution quality is affected by their cognitive styles [11].

In prior work, KABOOM has been used to simulate the performance of teams and agents with homogeneous and heterogeneous cognitive styles on two contextualized design problems [37]. For the first design problem, it was observed that changing the cognitive style of the homogeneous teams did not impact performance. For the second design problem, more innovative homogeneous teams outperformed the adaptive and mid-range teams. Overall, results demonstrated that some cognitive styles were more effective, depending on the problem. In another study, homogeneous teams of more adaptive, more innovative, and mid-range styles were tested on a set of 25 design problems to determine the optimal cognitive style for each problem. Larger search spaces favored more innovative cognitive styles, while smaller search spaces favored more adaptive styles [11]. When applied in different scenarios, KABOOM accurately models the trends found in A-I theory. For example, KABOOM emulates the communication styles of teams arranged in different network structures [38] and has exemplified the performance of specialized software development teams [39]. Further, it has been shown to represent trends in communication [40,41] and specialization [42] in human problem-solving.

We use the python implementation of the KABOOM framework in this work to generate teams of computational agents with varying simulated cognitive styles. This approach will evaluate how the agent composition and structure impacts robustness. By using KABOOM to study teams, we can rapidly simulate approximately 5000 design teams in this work, giving insight into potential strategies for team organization. Replicating this experiment using human subjects would be time and resource-intensive for both academic and industry research. KABOOM also enables detailed control over individual agents’ cognitive styles and over team composition—a level of control that would be challenging to achieve in human subjects research. By simulating thousands of teams, organizations will be able to predict the performance of the team before expending resources to assemble one. This leads to significant savings in cost, time, and productivity.

At the intersection of cognitive style and agent-based modeling, this research aims to determine how team composition and structure impact the robustness of engineering design teams. In prior research, it was found that varying the composition and hierarchal structure of design teams can affect their collaboration and problem-solving performance [9,14,16,18,19,22]. Therefore, we introduce two research questions that seek to understand how team robustness is affected by variations in cognitive style composition and structural hierarchy, respectively:

1. How does the robustness of team performance compare when homogeneous teams and heterogeneous teams solve the same set of design problems?

2. How does the addition of sub-teams affect the robustness of team performance in comparison to teams without sub-teams?

For this study, we used the KABOOM framework² to evaluate the impact of team composition (varying homogeneous and heterogeneous team types) and team structure (either with or without sub-teams) on robustness. As teams tend to naturally develop some degree of internal structure when larger than six members, we construct each team with six agents each [19]. Each team was tasked with solving the same set of 25 diverse design problems. By examining the distribution of performance across these problems, it becomes possible to assess how team structure and composition impact robustness. Every team is simulated for 300 iterations. Agents communicate with pairwise interactions occurring organically at a 20% frequency with each iteration. Agents that decide to communicate are paired and attempt to share their solutions.

In this work, the impact of team composition is examined through a variety of heterogeneous or homogeneous team compositions. In practice, KAI scores are addressed as an overall score, so sub-scores are not controlled for in this simulation. Homogeneous teams are composed of agents with the same overall KAI score, whereas heterogeneous teams are composed of agents with differing KAI scores. The simulated teams were composed using three different strategies. In the first composition strategy, homogeneous teams, all six agents in the team shared the same KAI score; seven different homogeneous team types were instantiated, linearly spaced between a score of 60 and 140. The second composition strategy composed of heterogeneous teams. These teams also included six agents, produced by sampling from the uniform distribution of the complete range of KAI scores observed in the population. This selection was iteratively narrowed to more mid-range scores, totaling five heterogeneous team types.

In addition to the teams mentioned above, we also simulated a single team with an organic composition. We compare the team composition strategies mentioned above to a team with an organic composition. The organic composition was generated by randomly selecting individuals from a virtual population corresponding to the normal distribution of KAI scores observed in the general human population (mean = 92.93, STD = 18.20) gathered in the earlier research [43]. The organic team-best represents a team selected without an overt strategy. Therefore, organic composition serves as a baseline against which to compare other approaches.

In addition to team composition, this work explores the impact of team structure (with or without sub-teams). Each problem that teams were tasked with solving contains 12 variables that can be decomposed by assigning subsets of the variables to different sub-teams, where each sub-team has unique control over their assigned variables, and assignments are not changed for the duration of the simulation [37]. We assess the impact of this team structure on both homogeneous and heterogeneous team compositions. In flat teams (those without sub-teams), the team collectively works on all 12 variables of the problem. In hierarchal (those with sub-teams) teams, each team was split into three sub-teams per team, where each sub-team controls a subset of all the dimensions in the problem. Three sub-teams per team were chosen as a representative value to emphasize problem specialization.

As teams often converge to analyze and evaluate solution ideas before moving forward in the presence of complex design problems [44], teams with sub-teams experienced team meetings. Team meetings occur every 30 iterations and result in all agents on the team converging to one solution. First, the team creates an aggregate solution from the specialized sub-teams. Each sub-team finds the best solution of any of its agents, then the team forms an aggregate solution of from the specialized dimensions of that sub-team’s best solution [11].

For every combination of team composition and team structure, 200 independent team simulations were performed (25 problems with eight repetitions each). At the end of each of these simulations, the team’s performance is the solution value of the best solution any agent has had at any time during the simulation, quantified as a numerical value. As the teams are solving a minimization problem in the simulation, lower numerical values indicate a better performance. We examine three values computed from the set of solution values for each combination. These include the median value, the lower quartile or 25th percentile value, and the worst value (after removing outliers). The median value indicates the median performance of the team. The 25th percentile value is calculated as the upper quartile (75th) value but presented as the lower quartile in the simulation to represent poorer performance in our minimization function. The worst score for each team represents the maximum score, excluding outliers. Outliers represent any value that exceeds the value calculated from the interquartile range multiplied by 1.5 [45]. In such cases, the latter value was presented as the worst score. In this work, high robustness in a team indicates their ability to reduce worst-case outcomes, achieving performance across problems with less detrimental variability. High robustness in a team indicates their propensity for performing consistently nearer to their median performance. A broad spread across these values indicates lower robustness (high dependence of solution quality on problem type), while a low spread indicates higher robustness (low dependence on problem type).

In addition to these values, the median, 25th percentile, and worst performance of the organic team are presented as a baseline against the strategized teams.

In this section, we discuss the robustness and median performance of simulated teams as a function of hierarchy and cognitive style. A Shapiro–Wilk test to assess normality indicated a significant departure from normality in our data. Therefore, multiple Mann–Whitney U tests were used to quantify differences in the objective function values between the various team compositions and structures. The Mann–Whitney U is a nonparametric test that can be used to determine if there are differences between two groups on a continuous scale [46].

The results of the strategized and organic team are presented in the figures below. Each figure shows the median, 25th percentile, and worst-case performance for each team composition. The error bars represent median absolute deviation, or the measure of variability around each data point. Horizontal dashed lines show the median, 25th percentile, and worst-case performance of an organically composed team as a baseline comparison. The median absolute deviation for the organic teams is not presented in the plots. As the agents are tasked with minimization problems, lower objective function values indicate better performance. We compare the trends with respect to robustness and overall median performance for each strategy and present three comparisons in the subsections. The first subsection addresses our first research question, how does the robustness of team performance compare when homogeneous teams and heterogeneous teams solve the same set of design problems? and presents the results of the homogeneous flat and heterogeneous flat teams. The second subsection addresses our second research question, how does the addition of sub-teams affect the robustness of team performance in comparison to teams without sub-teams presents the comparison between the flat and hierarchal teams.

The results of our homogeneous and heterogeneous flat teams are presented in Fig. 1.

A Mann–Whitney U test was run to evaluate if there exists a difference in the aggregate results of the performance between homogeneous and heterogeneous flat teams (those without sub-teams). Scores from each team in the homogeneous flat composition were aggregated and compared against the aggregate scores from the heterogeneous flat composition. Statistically significant differences were found between the aggregate values of the two groups U(N_homogeneous = 1400 N_{heterogeneous} = 1000) = 596,208, p < 0.001. This indicates that there are significant differences in the solutions outputted by either composition. Upon observation, as KAI scores for the homogeneous teams became more innovative, worst-scores became more severe with higher median absolute deviations. This indicates that as KAI scores increase, teams become less robust. In contrast, the heterogeneous teams performed more consistently than homogeneous teams, with lower variability for the median, 25th percentile, and worse-case scores. This indicates that heterogeneity in teams can help improve robustness.

The trend in the data observed in the homogeneous flat teams illustrates that higher KAI scores were positively correlated to better median performance. However, the most innovative team (KAI 140) saw a decrement in median performance. More research needs to be conducted to determine if median performance continues to decrement as team KAI scores increase past 140. In contrast to median performance, homogeneous flat teams only experienced better 25th percentile and worst-case scores for the more adaptive teams, specifically teams below a KAI of 100. In summary, slightly more innovative teams have a better median performance when faced with various problems compared to adaptive or highly innovative teams but are less robust as they are at a greater risk for outputting poor solutions. As more innovative individuals are known to think in risky and unexpected ways, these results seem to mimic real-life behavior commensurate with the A-I theory [47]. Finally, homogeneous teams showed a high degree of variability in their median performance, depending on the specific cognitive style. However, the performance of all heterogeneous teams aligned closely with that of the baseline organic teams.

Figure 2 illustrates the side-by-side results of homogeneous flat teams and homogeneous hierarchal teams. The earlier illustration for the homogeneous flat teams is provided as a comparison to the homogeneous hierarchal teams.

A Mann–Whitney test found that the differences in the distribution of the aggregate scores between homogeneous flat and hierarchal composition strategies were statistically significant U(N_flat = 1400 N_sub-team = 1400) = 882,149, p < 0.001. This indicates that there are significant differences in the solutions outputted by the hierarchal composition. In comparison to the homogeneous flat teams, the addition of sub-teams allowed for a smaller median absolute deviation, providing more consistency and a smaller spread of median, 25th percentile, and worst-case scores, especially for the more innovative teams. For more innovative teams, sub-teams greatly increased performance among the 25th percentile and worst-case scores. Specifically, hierarchal teams with a KAI score of 100 or higher were more robust than the flat teams, as indicated by the smaller spread of scores. Forming sub-teams in homogeneous team composition also helped to increase median performance for every team type except for our most adaptive teams (KAI score = 60), which did not benefit from the formation of sub-teams. Mann–Whitney U tests showed that for certain teams, these differences in median performance were statistically significant. Hierarchal teams with a KAI score of 100 had a significantly better median performance than corresponding flat teams with a KAI score of 100 U(N_flat = 200 N_sub-team = 200) = 15904, p = 0.001. Similarly, hierarchal teams with a KAI score of 113 outperformed a corresponding flat team with a KAI score of 113. Mann–Whitney confirmed that this difference was statistically significant U(N_flat = 200 N_sub-team = 200) = 17294, p = 0.009. The differences in median performance between the remainder of the teams were not found to be statistically significant. This demonstrates that teams that are slightly more innovative benefit significantly in median performance when organized into specialized sub-teams.

The results indicate that forming homogeneous teams with specialized sub-teams can greatly improve robustness, especially for more mid-range and innovative teams. Future work will evaluate if these results remain consistent with varying degrees of specialization. In sum, modifying the organization within the team can have significant implications on robustness and median performance, even if the KAI composition of the teams remains the same. It is also of note that among both composition strategies, the most adaptive teams performed the poorest regardless of team structure.

Figure 3 illustrates the side-by-side results of heterogeneous flat teams and heterogeneous hierarchal teams. The earlier illustration for the heterogeneous flat teams is provided as a comparison to the heterogeneous hierarchal teams.

While the differences in both robustness and median performance are visually less apparent between the flat and hierarchal compositions, a Mann–Whitney U test showed that there exists a significant difference in the distribution of the aggregate scores U(N_flat = 1000 N_sub-team = 1000) = 426,603, p < 0.0001. When comparing the results of the individual teams, heterogeneous hierarchal teams had significantly better median scores than the heterogeneous flat teams, except for the team pooled from the full distribution of scores (100 ± 40). A Mann–Whitney U test confirmed that these differences were statistically significant (for 100 ± 30 U(N_flat = 200 N_hierarchal = 200) = 878, p < 0.0001, for 100 ± 20 U(N_flat = 200 N_hierarchal = 200) = 787, p < 0.0001, for 100 ± 10 U(N_flat = 200 N_hierarchal = 200) = 673, p < 0.0001, for 100 ± 5 U(N_flat = 200 N_hierarchal = 200) = 733, p < 0.0001). For the team pooled from the full distribution of scores (100 ± 40), implementing sub-teams helped to increase robustness, visually apparent by the smaller spread of scores for the hierarchal team. Clearly, with diverse teams, specialized sub-teams can help increase median performance. Comparison with homogeneous hierarchal teams versus heterogeneous hierarchal teams followed similar trends.

Throughout the simulation study, it becomes apparent that some combinations of team structure and composition achieve high robustness while forfeiting median performance; for instance, highly adaptive homogeneous flat teams. We hypothesize that the highly localized search space exploration from a highly adaptive team is insufficient for solving problems that would benefit from a more global exploration resulting in a lower overall median performance. Homogeneous adaptive teams do not explore radical solutions that could result in poor performance, but they also do not explore the solution space sufficiently to find high-quality solutions [11]. These trends emulate A-I theory, where adaptors tend to “dig deeply” within the solution space and are less likely to explore the breadth of the solution space [9,48]. Other teams achieve good median performance while forfeiting robustness; for instance, highly innovative homogeneous flat teams. We hypothesize that a highly innovative team’s global exploration of the search space is sufficient for all problems, however, larger step sizes result in solutions that are very distinct from one another [11]. This increases the frequency and severity of poorer solutions, resulting in lower robustness. This is consistent with the A-I theory, as innovators are more likely to generate ideas that “jump around” within a solution space [9,48]. This tradeoff between robustness and median performance is potentially important for informing early team composition and structure decisions.

In this study, we evaluated the effects of team composition and team structure on robustness and median performance through the lens of Kirton’s theory of cognitive style. While cognitive style can be a valuable avenue from which we can use to form strategic teams, other factors also determine team performance, such as skills, proficiency, and personality. In addition, other aspects of cognitive styles, such as coping behavior [4], are omitted in the model to quantitatively assess the direct impact of individual KAI scores in collaboration with a team. Our findings in this work are also limited to the specific model parameters. It is possible that altering the number of agents per team, number of sub-teams, or frequency of team meetings may provide additional insights. While this model validates the behavior of adaptors and innovators described in A-I theory, it is not a comprehensive model of human problem-solving. Rather, this experiment allows us to predict how teams will perform given a variety of design problems with respect to their cognitive style. Future work will seek to validate results with human subjects. With all simulations of complex real-world systems, validation is necessary to truly understand the applicability of the numerical results. Extensive validation of our research outcomes is a subject of future work.

This experiment demonstrates that composing teams based on individual cognitive style may have significant impacts on a team’s ability to be robust given a variety of design problems. It also reveals that teams can be composed based on cognitive style in such a way that they outperform an organic team. Our first research question sought to evaluate the differences in robustness between homogeneous and heterogeneous team compositions. In this experiment, we concluded that homogeneous flat teams begin to forfeit robustness as their cognitive style composition becomes more innovative, while heterogeneous teams maintain robustness across the specific team compositions. Our second research question further examined how decomposing teams into specialized sub-teams further impacts robustness. We found that with the homogeneous teams, specialization can help reduce the severity of worst-case performance and thus increase robustness. Implementing hierarchy in heterogeneous teams also proves advantageous for median performance, as some teams median scores improved when decomposed into sub-teams.

In time, the combination of human subject studies and agent-based modeling methodologies will deepen our understanding of cognitive style and its implications for both individual and team problem-solving behavior. In this initial foray, we found that some combinations of team structure and composition sacrifice robustness for median performance, and while others sacrifice median performance for robustness. This tradeoff is vital to team success and should be informed by the team’s overarching objectives.

The first author’s research assistantship is supported by the Defense Advanced Research Projects Agency through cooperative agreement N66001-17-4064. Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the sponsor.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.